10 Installing the Oracle Big Data Appliance Software

This chapter explains how to install, reinstall, and reconfigure the software on Oracle Big Data Appliance. It contains these sections:

• About the Mammoth Utility

• Installation Prerequisites

• Downloading the Mammoth Software Deployment Bundle

• Installing the Software on a New Rack

• Adding Servers to a Cluster

• What If an Error Occurs During the Installation?

• Upgrading the Software on Oracle Big Data Appliance

• Enabling Optional Software Included With the Release

• Reinstalling the Base Image

• Installing a One-Off Patch

• Mammoth Software Installation and Configuration Utility

• Oracle Big Data Appliance Base Imaging Utilities

Note:

If you did not enter the required passwords in the Oracle Big Data Appliance Configuration Generation Utility, then you are prompted to enter them during the software installation. Ensure that you know the current passwords for the operating system root and oracle users, the Cloudera Manager admin user, and the MySQL administrator. If you are installing or reinstalling Oracle Big Data Connectors, then you also need the MySQL password for Oracle Data Integrator.

10.1 About the Mammoth Utility

Mammoth is a command-line utility for installing and configuring the Oracle Big Data Appliance software. Using Mammoth, you can:

• Set up a cluster for either CDH or Oracle NoSQL Database.

• Create a cluster on one or more racks.

• Create multiple clusters on an Oracle Big Data Appliance rack.

• Extend a cluster to new servers on the same rack or a new rack.

• Update a cluster with new software.

10.1.1 About Rolling Upgrades

A rolling upgrade is an upgrade method that avoids cluster downtime.

A rolling upgrade reboots the nodes of a cluster in a sequence that always leaves one node out of three replicas active. This allows the cluster to stay on-line throughout the entire series of reboots. As a result, some (though not all) services of the cluster can remain accessible without interruption during the upgrade. There are brief interruptions for non-redundant services such as HUE, Oozie, and HiveServer2 while the nodes where they run are rebooted. Also, when the MySQL master node is rebooted, Cloudera Manager, Hive, Oozie, and HUE are be briefly interrupted.

Note:

In Big Data Appliance 5.2 rolling upgrades are supported from release 5.1 to 5.2 only. Rolling upgrades to release 5.2 are not supported from CDH 5-based releases (Big Data Appliance 4.14 and earlier).

10.2 Installation Prerequisites

The Oracle Enterprise Manager options require software on a remote server on the same network as Oracle Big Data Appliance. Those installations must be complete before you run Mammoth, or it will fail.

Similarly, you must complete several steps for Kerberos if the key distribution center (KDC) is installed on a remote server. There are no preliminary steps if you install the KDC on Oracle Big Data Appliance.

The following list describes the prerequisites for all installation options.

Auto Service Request Requirements:

• Your My Oracle Support account is set up.

• ASR Manager is up and running.

Enterprise Manager Requirements:

• Oracle Management System (OMS) version 12.1.0.4.0 or higher is up and running.

• The OMS agent pull URL is working.

• The OMS emcli download URL is working.

• Both the HTTPS upload port and the console port are open.

Note:

Double-check the OMS credentials and ensure that you enter them correctly when running Mammoth. Invalid credentials are the primary cause of failure in the Enterprise Manager discovery process.

MIT Kerberos Requirements for a Remote KDC:

1. Add cloudera-scm/admin as a user to the KDC database by running the following command from kadmin:

   addprinc -randkey cloudera-scm/admin@<REALM NAME>
   

2. Grant cloudera-scm/admin all permissions to the Kerberos database. It must be able to add, modify, remove, and list principals from the database.

3. Create the cmf.keytab file by running the following command from kadmin:

   xst -k cmf.keytab cloudera-scm/admin@<REALM NAME> 
   

4. Move cmf.keytab to/opt/oracle/BDAMammoth.

5. To support Oozie and Hue, ensure that the remote KDC supports renewable tickets. If it does not, then follow these steps:

   1. Open kdc.conf and set values for max_life and max_renewable_life. The max_life parameter defines the time period when the ticket is valid, and the max_renewable_life parameter defines the time period when users can renew a ticket.

   2. Set maxrenewlife for the krbtgt principal. Use the following kadmin command, replacing duration with the time period and REALM NAME with the name of the realm:

      modprinc -maxrenewlife duration krbtgt/REALM NAME
      

   If the KDC does not support renewable tickets when Kerberos is configured, then Oozie and Hue might not work correctly.

Active Directory Kerberos Requirements for a Remote KDC

Active Directory Kerberos requirements are documented in MOS (My Oracle Support).

See MOS documents 2013585.1 and 2029378.1.

10.3 Downloading the Mammoth Software Deployment Bundle

The Mammoth bundle contains the installation files. Before you install the software, you must use Oracle Big Data Appliance Configuration Generation Utility to generate the configuration files, as described in "Generating the Configuration Files."

You use the same bundle for all procedures described in this chapter, regardless of rack size, and whether you are creating CDH or Oracle NoSQL Database clusters, or upgrading existing clusters.

Note:

The Mammoth software deployment bundle no longer includes a copy of the base image ISO. If you need the base image at a later time, you can find the download link in the My Oracle Support note referenced below – Oracle Big Data Appliance Patch Set Master Note (Doc ID 1485745.1)

To download the Mammoth bundle:

1. Locate the download site in either My Oracle Support or Automated Release Updates (ARU):

   My Oracle Support

   1. Log on to My Oracle Support and view Oracle Big Data Appliance Patch Set Master Note (Doc ID 1485745.1)

   2. From the list of Mammoth bundles, click the link to the bundle software version for this release.

   3. Find the link to the appropriate Mammoth patch bundle in the table.

   ARU

   1. Connect to ARU.

   2. On the Patches page, set Product to Big Data Appliance Integrated Software and Release to the appropriate release number.

   3. Click Search.

2. Download the BDA Mammoth ZIP files to any directory (such as /tmp) in the first node of the cluster. Depending on the configuration of the rack, this node can be the first, seventh, or thirteenth server from the bottom of the rack. For multirack clusters, this server is in the primary rack.

   The patch consists of a set of files labeled as follows:p<patch_number>_Linux-x86-64_<file number>of<total number of files>.zip

3. Log in to the first node of the cluster as root.

4. Extract all files from the all of the downloaded zip files. For example:

   $ unzip p<patch_number>_Linux-x86-64_1of4.zip
   	Archive:  p<patch_number>_<version>_Linux-x86-64_1of4.zip
     inflating: README.txt
      creating: BDAMammoth-ol7-<version>/
    ...
   

5. Change to the BDAMammoth-<version> directory:

   # cd BDAMammoth-ol7-<version>
   

6. Extract all files from BDAMammoth-<version>.run:

   # ./BDAMammoth-ol7-<version>.run
    
   Big Data Appliance Mammoth v<version> Self-extraction
   
   Checking for and testing BDA Package in /tmp
    
   Removing existing temporary files
    
   Generating /tmp/BDAMammoth.tar
   Verifying MD5 sum of /tmp/BDAMammoth.tar
   /tmp/BDAMammoth.tar MD5 checksum matches
    
   Extracting /tmp/BDAMammoth.tar to /opt/oracle/BDAMammoth
    
   Extracting Package RPMs to bdarepo
   Moving BDAPackages into /opt/oracle/
   
   Extracting BDA Extras
    
   Removing temporary files
        .
        .
        .
   Please "cd /opt/oracle/BDAMammoth" before running "./mammoth -i <rack_name>"
   #
   

   The new version of the Mammoth software is installed in /opt/oracle/BDAMammoth, and the previous version (if you are upgrading) is saved in /opt/oracle/BDAMammoth/previous-BDAMammoth.

7. Follow the specific instructions for the type of installation you are performing.

10.4 Installing the Software on a New Rack

The Mammoth utility installs and configures the software on Oracle Big Data Appliance using the files generated by Oracle Big Data Appliance Configuration Generation Utility. A cluster can be dedicated to either CDH (Hadoop) or Oracle NoSQL Database. Note that CDH and Oracle NoSQL Database cannot operate within the same cluster.

For a CDH cluster, Mammoth installs and configures Cloudera's Distribution including Apache Hadoop. This includes all the Hadoop software and Cloudera Manager, which is the tool for administering your Hadoop cluster. .

For a NoSQL cluster, Mammoth installs Oracle NoSQL Database.

Mammoth can also install some optional Oracle software such as Oracle Big Data Connectors and Oracle Big Data SQL.

In addition to installing the software across all servers in the rack, Mammoth creates the required user accounts, sets the appropriate configuration parameters and starts the services on the cluster.

10.4.1 Installing the Software

Follow this procedure to install and configure the software on one or more Oracle Big Data Appliance racks. You can configure one cluster on multiple racks in a single installation.

To install the software:

1. Verify that the Oracle Big Data Appliance rack is configured according to the custom network settings described in /opt/oracle/bda/network.json (or, cluster-network.json). If the rack is still configured to the factory default IP addresses, first perform the network configuration steps described in "Configuring the Network."

2. Verify that the software is not installed on the rack already. If the software is installed and you want to upgrade it, then use the mammoth -p option in Step 6.

3. Download and unzip the Mammoth bundle. You must be logged in as root to the first server in the cluster.

4. Change to the BDAMammoth directory.

   # cd /opt/oracle/BDAMammoth
   

5. Copy cluster_name-config.json to the current directory.

6. Run the mammoth command with the appropriate options. This sample command runs all steps:

   ./mammoth -i rack_name
   

   After Mammoth completes Step 3 of the installation, it prompts you to restart, if it upgraded the base image.

7. If you installed support for Auto Service Request, then complete the steps in "Verifying the Auto Service Request Installation."

8. To configure another CDH cluster on the server:

   1. Copy the BDAMammoth ZIP file to any directory on the first server of the cluster, which is either server 7 or server 13.

   2. Repeat Steps 3 to 7. Each cluster has its own cluster_name-config.json file. Oracle Big Data Appliance Configuration Generation Utility creates the files in separate directories named for the clusters.

See Also:

• "Downloading the Mammoth Software Deployment Bundle"

• "About the Configuration Files"

• "Mammoth Software Installation and Configuration Utility."

Note:

Mammoth stores the current configuration in the /opt/oracle/bda/install/state directory. Do not delete the files in this directory. Mammoth fails without this information if you must use it again, such as adding a rack to the cluster.

Verifying the Auto Service Request Installation

1. Log in to My Oracle Support at https://support.oracle.com/portal.

2. Search for document ID 2103715.1, Engineered Systems ASR Configuration Check tool (asrexacheck version 4.x), and download the asrexacheck script.

   Although this check was originally intended for Oracle Exadata Database Machine, it is now supported on Oracle Big Data Appliance.

3. Copy asrexacheck to a server in the Oracle Big Data Appliance cluster.

4. Log in to the server as root.

5. Copy asrexacheck to all servers in the cluster:

   # dcli -C -f asrexacheck -d /opt/oracle.SupportTools
   

6. Change the file access permissions:

   # dcli -C chmod 755 /opt/oracle.SupportTools/asrexacheck
   

7. Run the script:

   # dcli -C /opt/oracle.SupportTools/asrexacheck
   

8. File an Oracle Support Request (SR) to validate the ASR installation. Note the following choices:

   1. Under "What is the Problem?" click the Hardware tab.

   2. For Products Grouped By, select Hardware License.

   3. For Problem Type, select My - Auto Service Request (ASR) Installation and Configuration Issues.

   4. Include the output of asrexacheck.

9. Go back to previous set of steps (To Install the Software:) and continue with Step8.

See Also:

• For information about dcli:

  Executing Commands Across a Cluster Using the dcli Utility

10.5 What If an Error Occurs During the Installation?

If the Mammoth utility fails, take these steps to resolve the problem:

1. Read the error message to see if it suggests a cause or a resolution.
2. Make the recommended changes and rerun the step.
3. If the error message does not recommend a resolution, or it does not resolve the problem:

   • File a service request (SR) with My Oracle Support.

   • Upload the diagnostic zip file, which Mammoth generates when an error occurs.

See Also:

Oracle Big Data Appliance Software User's Guide

10.6 Adding Servers to a Cluster

You can use the Mammoth installer to add servers to a cluster.

To add servers to an existing cluster, use the Mammoth Utility to extend the cluster onto the new servers. Do not use the Oracle Big Data Appliance Configuration Generation Utility for this purpose.

Note:

• A single-rack cluster must always include a minimum of three nodes.
• If the cluster is extended to a second rack, cluster, then the first rack must include a minimum of four nodes.
• The second rack and all subsequent racks within a multi-rack cluster may contribute one to any number of nodes to the cluster.

There is no cluster downtime in this process for either single or multi-rack clusters.

It is not necessary to decommission edge nodes when adding new nodes to an existing cluster.

There is no significant difference between how new server nodes and old nodes are configured. However, when a new node has a larger disk capacity, the data partitions created are correspondingly larger in order to use the entire disk. HDFS handles the fact that some data nodes have more space available than other nodes. Oracle Big Data Appliance performs no additional special configuration.

See Also:

• "About Rolling Upgrades"

To install the software on additional servers in a cluster:

Before You Start:

• See "Mammoth Software Installation and Configuration Utility" for an explanation of the Mammoth -e option.

1. Ensure that all servers are running the same software version. The additional servers must not have an Oracle Big Data Appliance base image that is newer than the existing cluster.

2. Ensure that all racks that form a single Hadoop cluster are cabled together.

3. Connect as root to Node1 of the primary rack and change to the BDAMammoth directory:

   cd /opt/oracle/BDAMammoth
   

   Note: Always start Mammoth from the primary rack.

4. Generate a parameter file for the server group. The following example adds six servers beginning with Node13:

   ./mammoth -e node13 node14 node15 node16 node17 node18
   

   You can also use the -r (range) or -s (step) options in conjunction with mammoth -e, as in:

   # mammoth -r 1-4 -e newnode1 newnode2 newnode3...

   # mammoth -s 2 -e newnode1 newnode2 newnode3...

   The servers can be on the same rack or multiple racks.

   After Mammoth completes Step 3 of the installation, it prompts you to restart, if it upgraded the base image.

5. If you are using Oracle Enterprise Manager Cloud Control to monitor Oracle Big Data Appliance, then run rediscovery to identify the hardware and software changes.

If you have a license for Oracle Big Data Connectors, then they are installed on all nodes of the non-primary racks, although the services do not run on them. Oracle Data Integrator agent still runs on Node3 of the primary rack.

Mammoth obtains the current configuration from the files stored in /opt/oracle/bda/install/state. If those files are missing or if any of the services have been moved manually to run on other nodes, then Mammoth fails.

See Also:

• "About Software Version Differences"

• "Connecting Multiple Oracle Big Data Appliance Racks"

About Software Version Differences

All servers configured as one Hadoop cluster must have the same image. A new Oracle Big Data Appliance rack might be factory-installed with a newer base image than the previously installed racks. Use the imageinfo utility on any server to get the image version. When all servers of a single Hadoop cluster have the same image version, you can install the software.

To synchronize the new servers with the rest of the Hadoop cluster, either upgrade the existing cluster to the latest image version or downgrade the image version of the new servers.

To upgrade the image version:

• Run Mammoth with the -p option to upgrade the software on the cluster to the latest version. See "Upgrading the Software on Oracle Big Data Appliance."

To downgrade the image version:

• Reimage the new rack to the older version installed on the cluster.

• Use the old version of the Mammoth utility, which is on the first server of the existing cluster, to extend the cluster onto the new rack.

See Also:

See Oracle Big Data Appliance Installation Frequently Asked Questions (FAQ) (Doc ID 1518939.1) for information about reimaging. Oracle Big Data Appliance Frequently Asked Questios (DOC ID 1518939.1)

10.7 Upgrading the Software on Oracle Big Data Appliance

The procedure for upgrading the software is the same whether you are upgrading from one major release to another or just applying a patch set. The procedure is also the same whether your Hadoop cluster consists of one Oracle Big Data Appliance rack or multiple racks.

The process upgrades all components of the software stack including the firmware, Oracle Linux Unbreakable Enterprise Kernel (UEK), CDH, JDK, and Oracle Big Data Connectors (if previously installed).

To upgrade only Oracle Big Data Connectors, and no other components of the software stack, contact Oracle Support for assistance.

Software downgrades are not supported.

Note:

The cluster will be unavailable while the mammoth command is executing.

If you are upgrading from a release that did not included high availability Sentry, Mammoth displays a message to notify you that enabling HA for Sentry requires a short shutdown of all Sentry-dependent services.

Currently, Oracle Big Data Appliance software upgrades do not require corresponding upgrades of the InfiniBand or Cisco switches.

10.7.1 About the Operating System Versions

This Oracle Big Data Appliance release supports both Oracle Linux 7 for new installations and Oracle Linux 6 for upgrades. Oracle Linux 5 is not supported.

10.7.2 Upgrading the Software

Follow the procedures in this section to upgrade the software on an Oracle Big Data Appliance cluster to the current version.

You can upgrade to Oracle Big Data Appliance Release 5.2 directly from Releases 4.12, 14.13, 4.14, and 5.1. See the Cloudera Enterprise Upgrade Guide for supported Cloudera upgrade paths.

Note:

In Big Data Appliance 5.2, rolling upgrades (which avoid cluster downtime) are supported from Big Data Appliance 5.1 to 5.2 only. Upgrades from earlier releases will require some downtime.

See Also:

Before you being the upgrade steps, review the Known Upgrade Issues

Can I Upgrade CDH Manually, Outside of the Mammoth Upgrade?

Upgrades to newer CDH major releases are not supported outside of Mammoth. These must be done as part of a full-stack Mammoth upgrade. For example, it is not supported to manually upgrade a CDH 6.2.1 environment to CDH 6.3.4. Nor could can you manually upgrade a release such as CDH 5.11 to 6.2.1.

Minor (third-digit) CDH upgrades to newer maintenance releases within the same major release are supported. For example, you can manually upgrade CDH 5.11 to 5.11.2

10.7.2.1 Pre-Upgrade Checklist

Note:

If you are using Key Trustee Servers that are external to the Oracle Big Data Appliance, you should upgrade the servers where the external Key Trustee Servers are configured before upgrading Oracle Big Data Appliance. After the Key Trustee Servers are upgraded, you can work through the Oracle Big Data Appliance cluster upgrade prerequisites and then upgrade the appliance.

You must also import the certificates from the external Key Trustee servers. The procedure to do this is included in the steps below.

1. Obtain the list of passwords currently in effect for the cluster. Mammoth will prompt you for these:

   • oracle

   • root

   • Cloudera Manager admin

   • MySQL Database admin

2. Disable Oracle Big Data Discovery. This product is not supported in Oracle Big Data Appliance 5.2. The ODI Agent must not be present when you run the upgrade. Uninstall it before upgrading.

   # bdacli disable bdd

3. Uninstall the ODI agent. The Mammoth installation in this release does not install the ODI agent and you will not be able to install the agent via bdacli after the Mammoth installation.

   There is no separate uninstall procedure for the ODI agent. To remove it, uninstall Oracle Big Data Connectors and then reinstall Oracle Big Data Connectors without the agent. You will need to authenticate with the Configuration Manager admin password and the MySQL root password for both operations.

   # bdacli disable bdc

   # bdacli enable bdc

   When you enable Big Data Connectors, exclude the ODI agent by responding n to this prompt:

   Do you wish to enable ODI? [y/n]: n

   After the Mammoth installation, you can separately install the standalone ODI Agent version 12.2.1.4 using the instructions in the ODI product documentation. This version of the agent is certified as compatible with Cloudera 6.

4. Stop and remove the following services before the upgrade. These are no longer supported as of Cloudera Enterprise 6.

   Accumulo
   Sqoop 2
   MapReduce 1
   Record Service

   Spark 1.6 is also not supported, but it is automatically removed during the upgrade.

   In Oracle Big Data Appliance 4.1, Apache Sentry changed from a file-based provider to the Sentry service. If Sentry as a file-based provider was enabled when you upgrade the cluster, then the file-based provider is retained. To switch to the Sentry service, disable Sentry before you upgrade the software, and then re-enable Sentry afterward.

5. If the cluster has a Solr service, check to ensure that at least one collection exists in that Solr service. Otherwise, either add a collection or remove the service.
6. Stop all Oozie jobs before the upgrade. Failure to do this may cause the upgrade to fail.
7. If you are using external Key Trustee Servers, you should import the external Key Trustee Servers' certificates before starting the upgrade, as follows.

   For the active Key Trustee Server:

   1. Copy the key-trustee certificate /opt/cloudera/security/x509/hostname.pem from the active Key Trustee Server to the /root directory on the Mammoth node (the node where Mammoth runs, which is usually node1).
   2. On the Mammoth node, run keytool -importcert -file /root/hostname.pem -keystore /opt/cloudera/security/jks/<cluster_name>.truststore -alias keytrustee_active.

      Be sure to include the full path to the truststore. Otherwise, the command will create a new truststore instead of modifying the existing one.

   For the passive Key Trustee Server:

   1. Copy the key-trustee certificate /opt/cloudera/security/x509/hostname.pem from the active Key Trustee Server to the /root directory on the Mammoth node.
   2. On the Mammoth node, run # keytool -importcert -file /root/hostname.pem -keystore /opt/cloudera/security/jks/<cluster_name>.truststore -alias keytrustee_passive.

      Be sure to include the full path to the truststore. Otherwise, the command will create a new truststore instead of modifying the existing one.

   3. On the Mammoth node, run # dcli -C -f /opt/cloudera/security/jks/<cluster_name>.truststore -d /opt/cloudera/security/jks
8. If the cluster has KMS and HBase master roles in the same node, you must move the HBase master role to another node. If the KMS and HBase master roles are in the same node an error will occur during the upgrade because both roles are using the same port (16000).
9. If Big Data SQL is installed on the cluster and the cluster is running Oracle Linux 6 (not Oracle Linux 7), then install the Python cryptography module if it is not already present.

   On the node where Cloudera Manager runs (usually node3):

   1. Download the get-pip.py script.

      curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

   2. Install pip, using scl:

      scl enable python27 'python get-pip.py'

   3. Install the Python cryptography module:

      scl enable python27 'pip install cryptography'

      Tip:

      To verify that the module was installed you can run: scl enable python27 'python -c "import cryptography"' and then check that the return code is 0.

10.7.2.2 Using the Mammoth "step" and "range" Options in Upgrades

Several Mammoth installer options are supported for use with upgrades.

The Mammoth "step" (-s) and "range" (-r) parameters are options that can be used in combination with the standard mammoth -p command.

• mammoth -s <n> -p

  Run the specified Mammoth upgrade step only.

• mammoth -r <n>-<n> -p

  Run Mammoth upgrade steps in the range of n through n

One situation in which these options can be useful is when the maintenance window available for upgrade may be insufficient to run a complete upgrade. You can safely complete a single upgrade step or range of steps and then resume the upgrade at a later time.

The table below shows the upgrade steps. Provide the appropriate step number values to -s or -r in a Mammoth upgrade command.

Table 10-1 Upgrade Steps

Step	Description
1	Pre-checks and pre-configuration for upgrade
2	Operating system upgrade
3 and 4	Cloudera Manager upgrade
5	CDH upgrade
6	Big Data Appliance software upgrade
7	Start Cloudera services
8	Start Cloudera services

Examples

• Run the pre-checks and upgrade the OS only:

  # mammoth -r 1-2 -p

• Uprade Cloudera Manager only:

  # mammoth -r 3-4 -p 

• Upgrade CDH only:

  # mammoth -s 5 -p

• Start Cloudera services:

  # mammoth -s 7 -p

• Upgrade Big Data Manager and do final cleanup:

  # mammoth -s 8 -p    

10.7.2.3 Upgrading to the Current Software Version

Note:

It is important that all services are healthy before upgrade, especially after a reboot.  Check Cloudera Manager to ensure that the status indicators for all services are "green" (in good health) before proceeding.

Also note that Kafka clusters may be upgraded (and extended) in this release. This functionality was not available in Big Data Appliance 4.14 and 5.1.

Upgrade the Oracle Big Data Appliance software to the current software version as follows. This is a summary. Refer to Doc ID 2101906. in My Oracle Support for details, including prerequisites, further information on the steps below, and known issues.

1. Download and unzip the Mammoth bundle, as described in "Downloading the Mammoth Software Deployment Bundle." You must be logged in as root to node 1 of the primary rack.
2. Change to the BDAMammoth directory.

   # cd /opt/oracle/BDAMammoth
   

3. How you start the upgrade depends on whether or not Solr is installed within the cluster.
   • If there is no Solr service in the cluster, then start the upgrade with the mammoth-p command :

     # ./mammoth -p

   • If a Solr service is present, run the first four steps using the Mammoth range option (-r) to stop temporarily after Step 4.

     # mammoth -p -r 1-4

     When Step 4 is complete and before proceeding further with the upgrade, go to http://tiny.cloudera.com/cm-cdh6-preupgrade-solr-6 and work through the Solr pre-upgrade steps documented by Cloudera.

     After you have completed the steps provided by Cloudera, continue with the Oracle Big Data Appliance upgrade without specifying a range.

     # ./mammoth -p

   Mammoth automatically upgrades the base image if necessary.

4. After all nodes reboot, perform the following checks.
   1. Check uptime.
      # dcli -C uptime
   2. Check /root/BDA_REBOOT_SUCCEEDED.
      # dcli -C ls -ltr /root/BDA_REBOOT_SUCCEEDED
      Note: If there is no BDA_REBOOT_SUCCEEDED, check for /root/BDA_REBOOT_* and /root/bda_reboot_status.
   3. Verify that the kernel and JDK are upgraded.

      # dcli -C uname -a 
      # dcli -C java -version

   4. Check that all Cloudera Configuration Manager services are healthy. You may need to manually restart some services.

      Note:

      During any upgrade, it is crucial that all services in CM are healthy after the reboot before you continue.  Unhealthy services may result in upgrade failures.

      Oracle Big Data SQL is exception to this. During an upgrade from any Oracle Big Data Appliance 4.x release to 5.2, if Big Data SQL is already installed, the BigDataSQL service in CM will initially show bad health after a reboot. This is expected and will be automatically corrected in every case later in Mammoth upgrade. You can opt to correct it immediately by using Big Data SQL's Jaguar utility as explained in Known Upgrade Issues.

5. After the reboot and the post reboot checks, log on to node 1 of the primary rack and rerun mammoth -p in order to resume the upgrade.

   # cd /opt/oracle/BDAMammoth
   # ./mammoth -p

6. When the upgrade is complete, perform the post-upgrade validation steps described in the MOS document (2101906.1).

10.7.2.4 Known Upgrade Issues

There are errors that may occur during the upgrade.

Failure to Disable Oracle Big Data Discovery or the ODI (Oracle Data Integrator) Agent

As mentioned in the section, there are products and services that must be disabled or removed prior to the upgrade. If you did not disable Oracle Big Data Discovery or the ODI agent prior to the upgrade, you will see this message during the upgrade:

BDA version and Mammoth version are not the same: BDA-5.2.0  Mammoth-4.12.0

Ordinarily, you would run bdacli commands to disable these components, but an attempt to do so in this situation returns one of these error message.

ERROR: Big Data Discovery is not supported for this BDA version.

ERROR: Oracle Data Integrator is not supported for this BDA version.

The workaround is to temporarily revert to the previous version of the Mammoth software and then use bdacli to disable Oracle Big Data Discovery and/or the ODI agent. When that is done, switch back to the newly installed Mammoth software. We can do this because the previous version of Mammoth is not deleted by the upgrade. It is stored at /opt/oracle/BDAMammoth/previous-BDAMammoth.

Follow this procedure:

1. Uninstall the new Oracle Big Data Appliance RPM that was installed by the upgrade. For example:

   # rpm -e bda-5.2.0-1.el6.x86_64

2. Move the BDAMammoth directory that was newly created by the upgrade to a backup location. For example:

   # mv /opt/oracle/BDAMammoth/ /opt/oracle/BDAMammoth_v5

3. Copy and rename the previous-BDAMammoth directory (the backup that Mammoth created) to /opt/oracle/BDAMammoth:

   # cp -pR /opt/oracle/BDAMammoth_v5/previous-BDAMammoth/ /opt/oracle/BDAMammoth

4. Change directories to /opt/oracle/BDAMammoth/bdarepo/RPMS/x86_64 and re-install the previous (pre-upgrade) Oracle Big Data Appliance RPM. For example:

   # rpm -ivh bda-4.12.0-2.el7.x86_64.rpm

5. Now try again to disable Oracle Big Data Discovery or the ODI agent:

   # bdacli disable bdd

   Note:

   There is no bdacli command to disable the ODI agent alone. You must uninstall Oracle Big Data Connectors and then reinstall it without the ODI agent. You can find the commands for this in the Pre-Upgrade Checklist
6. If the removal succeeds, then switch back to the new Mammoth software. First, delete the temporary installation of the old software at /opt/oracle/BDAMammoth and then move and rename the archive of the new directory back to /opt/oracle/BDAMammoth:

   # rm -rf /opt/oracle/BDAMammoth

   # mv /opt/oracle/BDAMammoth_v5 /opt/oracle/BDAMammoth 

You do not need to uninstall the old Big Data Appliance RPM and reinstall the newer one. The upgrade will do this. Now you can resume the upgrade:

# mammmoth -p

Error Reported When HBase and Kerberos Key Management Server are on the Same Node

As mentioned in the pre-upgrade checklist, in CDH 6, KMS and HBase by default both use port 16000. Therefore, if both are running on the same node, an "Address already in use" error will occur during the upgrade. You can see this error in the Cloudera Manager Admin Console. In the console, click the indicator that shows the number of running commands. Click All Recent Commands and search for the latest execution of the Upgrade Cluster command.

Look for this error message:

java.lang.RuntimeException: Failed construction of Master
Caused by: org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

To fix this error, change the HBase Master port. In Configuration Manager, navigate to HBase->configuration and change the value of hbase.master.port. Check that the port number you assign is not already in use.

Run mammoth -p to continue the upgrade after resolving this conflict.

Expected Error at Upgrade Step 5 (CDH Upgrade) if Solr is Already Installed on the Cluster

This error will always occur if CDH 5 Solr or earlier is present on the pre-upgrade system.

ERROR: This is an expected error. 
The Apache Solr version in Cloudera Search in CDH 5 is incompatible with the Solr version in CDH 6.
Please, follow the documentation to fix this issue before continue with the upgrade.

The error message refers to the Cloudera documentation. See Migrating Cloudera Search Configuration Before Upgrading to CDH 6. After completing the Cloudera instructions, run mammoth -p to continue.

Error Starting the YARN ResourceManager at Step 5

This intermittent error may be logged in Cloudera Manager under Recent commands->Upgrade Cluster.

Error starting ResourceManager
org.apache.hadoop.service.ServiceStateException: org.apache.zookeeper.KeeperException$NoAuthException: 
KeeperErrorCode = NoAuth for /yarn-leader-election/yarnRM/ActiveStandbyElectorLock

You can fix the problem by removing /yarn-leader-election and /rmstore/ZKRMStateRoot in ZooKeeper

To do this, add the super-auth key to ZooKeeper. You can use Java to obtain a digest that you then use to temporarily define a superuser within the ZooKeeper configuration options in Configuration Manager. You can then use that superuser to remove /yarn-leader-election and /rmstore/ZKRMStateRoot.

1. First, run this command to return the digest needed to enable the ZooKeeper superuser with the password of your choice:

   java -classpath zookeeper.jar:lib/* org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:cloudera

   The command returns the digest. For example: super:cloudera->super:cY+9eK20soteVC3fQ83SXDvwlP0=

2. In Configuration Manager, go to Zookeeper service configuration -> Java Configuration Options for ZooKeeper Server.
3. Append the following string to the configuration options. Important: Be sure that you do not overwrite any existing options.

   -Dzookeeper.DigestAuthenticationProvider.superDigest=<The required digest>

   For example, using the digest returned in the first step:

   -Dzookeeper.DigestAuthenticationProvider.superDigest=super:cY+9eK20soteVC3fQ83SXDvwlP0=

4. Save the changes and restart the ZooKeeper service.
5. Use the ZooKeeper client to SSH into one of the Hadoop nodes and run the three commands below:

   zookeeper-client -server zk_host:2181
   addauth digest super:cloudera
   rmr /yarn-leader-election
   rmr /rmstore/ZKRMStateRoot

6. Restart the YARN service.
7. Remove -Dzookeeper.DigestAuthenticationProvider.superDigest from Zookeeper service configuration -> Java Configuration Options for ZooKeeper Server.
8. Save your changes and then restart the ZooKeeper service.
9. Resume the upgrade:

   # mammoth -p

Tip:

These instructions consistently tell you to use mammoth -p to resume the upgrade. If you prefer, you can also resume the upgrade through Configuration Manager (In CM, go to All recent commands, find Upgrade CDH Cluster, and click the Resume option.) These methods are equivalent, but with mammoth -p you will be able to more quickly see any errors that may occur.

Configuration Manager Shows That BigDataSQL Health is Bad After Node Reboots

This issue occurs in every case if Oracle Big Data SQL is already installed prior to the upgrade. After the reboot of each node during the upgrade, Cloudera Configuration Manager will indicate that the health of the BigDataSQL service is bad. This is expected and occurs only because JAVA_HOME has changed. The correction occurs later in the upgrade process and BigDataSQL is then restored to good health. In some cases you may not want to wait for the automatic restore to good health, for example, if you are performing the upgrade across several maintenance windows. You can manually restore the BigDataSQL service to good health by using the Big Data SQL Jaguar utility. The Jaguar reconfigure operation corrects the problem.

# ./jaguar reconfigure <myconfigfilename>.json

The default path and configuration file is /opt/oracle/BDSjaguar/bds-config.json . However, the person who installed Big Data SQL may have set up a different path and used a different file name. Search for the BDSjaguar directory. You will find the configuration file there. No changes to the configuration file are required in order to fix the service health issue.

See Also:

About the Jaguar Utility in the Oracle Big Data SQL Installation Guide provides more information about the Jaguar utility and the reconfigure operation.

10.7.2.5 Known Post-Upgrade Issues

The following issues may arise after the upgrade.

Warning About Required Psycopg2 Version

If you run the Cloudera Manager Host Inspector after the upgrade, you will see the following warning:

Warning: Starting with CDH 6, PostgreSQL-backed Hue requires Psycopg2 version to be at least 2.5.4

This is expected. The warning is harmless, because Big Data Appliance clusters use a MySQL database, not PostgreSQL.

10.7.2.6 Performing Parcel Upgrades Outside of the One-Off Patch Mechanism

Oracle one-off patches for Oracle Big Data Appliance software are by definition supported. Parcel upgrades that are not part of a one-off patch are unsupported unless affirmed by Oracle Support.

Mammoth bundles include a recent version of the parcel for Cloudera's Distribution including Apache Hadoop (CDH). However, CDH is released on Cloudera’s schedule and sometimes a newer CDH release than the one in the current Mammoth bundle may be available. A new CDH version may be supported for integration with the current Oracle Big Data Appliance release. Consult with Oracle Support if you have questions.

10.8 Migrating from Oracle Linux 6 to Oracle Linux 7

If you want to extend existing Oracle Linux 6 clusters with X8-2L servers, then consider upgrading the cluster to Oracle Linux 7. X8-2L servers are shipped with Oracle Linux 7 installed. Oracle Linux 6 clusters do not support Oracle Linux 7 nodes.

This upgrade is for CDH clusters only. It does not apply to Oracle NoSQL Database clusters.

Note:

Migration after upgrade is preferred. Oracle Linux 6 to Oracle Linux 7 migration is supported in releases 4.13 and greater. For any of these releases you can perform a migration from Oracle Linux 6 to Oracle Linux 7 either before or after the upgrade to Big Data Appliance 5.2. However, in the case of Big Data Appliance 4.13, if you choose to migrate to Oracle Linux 7 it is recommended that you perform the migration after the upgrade to Big Data Appliance 5.2. This is is because some improvements to the process were introduced after release 4.13.

The Operating System Migration in This Release Requires Two Patches

1. Patch 32462909: ORACLE BIG DATA APPLIANCE MAMMOTH DEPLOYMENT BUNDLE V5.2.0 FOR ORACLE LINUX 7
2. The latest Oracle Big Data Appliance Base Image for Oracle Linux 7. This image includes support for migration from Oracle Linux 6 to Oracle Linux 7.

   https://support.oracle.com/epmos/faces/PatchDetail?patchId=28218768

Estimating Time Required for the Migration

The total time to perform the cluster-wide Oracle Big Data Appliance Mammoth upgrades and then the per-node OS migration can be substantial. The main factors are the size of the cluster and the server model of the nodes.

The migration may take three to four hours for a slave node. For a master node, the process may take five hours or more, depending on the size of the MySQL database.

About Migrating Multiple Nodes at the Same Time

It is not possible to migrate more than one critical node at a time. (Critical nodes are Node1 to Node4.)

It is possible to migrate two non-critical nodes (but no more that two) at the same time, with this constraint:

• For the first pair of nodes being migrated, the execution of bdarestorenode.sh must complete on the first node before you can start the run of bdarestorenode.sh on the second node.

In general, it is recommended to avoid running bdarestorenode.sh more than once at the same time within the cluster. However you can run bdadumpnode.sh, bdareimagenode.sh, and perform other steps in the process concurrently on two non-critical nodes.

Note:

While the migration is on-going, some nodes will be running on Oracle Linux 7 and others will still be running on Oracle Linux 6. It is safe to continue to use the cluster during this time. However, do not add new services to the cluster while the upgrade is in progress.

10.8.1 Important Considerations

Here are some things you should know before you start the upgrade.

The OS Upgrade Uses the Latest Oracle Big Data Appliance Base Image

You will need to download the latest Oracle Big Data Appliance base image for Oracle Linux 7 (patch 28218768), either directly to each node or to a location in the cluster where you can then copy the image to each node. In this process, you unzip the base image on each node, but do not install it. The OS upgrade leverages some functionality that is available in the base image.

Only Mammoth-Installed Software is Migrated

The OS upgrade process does not back up and does not migrate software that was not installed by Mammoth, the Big Data Appliance installation utility.

If you have installed third-party software products outside of the Mammoth installation mechanism, then you must make separate provisions for backing up and restoring or reinstalling such software after the upgrade. This includes Oracle software that is not installed by Mammoth as well as Cloudera software that may be included in the Mammoth software deployment bundle for your convenience, but is not installed by Mammoth (such as Kudu and Kafka).

Also, Oracle Linux modules added by the customer to Oracle Linux 6 kernel are not migrated to the Oracle Linux 7 installation.

On Each Node, Roles That are not Installed by Mammoth Must be Removed Prior to the Upgrade of the Node

Before starting the upgrade on each node, use Cloudera Manager to find the roles running on the node. Be sure to remove all roles not installed by Mammoth, otherwise the upgrade may not succeed.

File Backup Considerations

The process generates a list of essential files to be backed up. You can add your own list as well.

HDFS data is preserved in the upgrade.

It is the customer’s responsibility to separately back up any non-HDFS data that should be preserved prior to the migration. It is strongly recommended to back up all files that are not stored in HDFS.

Service Interruptions During the Upgrade

The upgrade of critical nodes (generally nodes 1 to 4) will necessarily include temporary interruptions of important services, but at no point does it shut down the entire cluster. Jobs that are in process can continue to run. Hadoop functionality remains partially or mostly available. The specific service outages at any point in the process depend upon which services and roles are running on the node that is currently undergoing upgrade.

Because of the three-factor replication of data on Oracle Big Data Appliance, there is no loss of service during the upgrade of non-critical nodes (nodes which primarily store data). Ensure that the replication factor is set to 3 (the default) on the cluster. If it is not, then data on a node undergoing the upgrade may be temporarily unavailable.

The distribution of services and roles across nodes of single-rack and multirack clusters include some differences. Since services running on a node are unavailable while the node is undergoing upgrade, this may be a factor in how you order and schedule the upgrade on the nodes of the cluster. Other than this, there are no special considerations in the upgrade process for single-rack versus multirack clusters.

Security Setups are Preserved

Kerberos security configurations as well as HDFS Transparent Encryption and HTTPS/ Network Encryption are preserved in the upgrade.

Cloudera Manager roles are retained in the upgrade.

The MySQL Database is Preserved

In the upgrade steps on each node, you run the bdadumpnode.sh script to preserve information on the node. On the node where the MySQL database runs (generally on Node3), this includes a dump of the MySQL database. If you want to do an additional dump of the database, then do so immediately before beginning the upgrade on Node3. Ensure that the database state in the manual dump is consistent with the automatic dump performed by the upgrade. Store the manual dump in a separate location on another node.

Run Health Checks After Each node Upgrade and Rebalance Loads if Needed

After the upgrade has completed on each node, recheck cluster health (mammoth -c) and via the Cloudera Manager web console recheck services health status.

During the OS upgrade, HDFS data on the node being upgraded is automatically replicated to other nodes. If the nodes receiving the replicated data are already under heavy load, then in CM, the replication may trigger a yellow “Concerning heath” indicator for some DataNode roles.

If this situation occurs, then before proceeding to the next upgrade run HDFS Balancer in CM. If over-utilization is the cause of the yellow warning indicator, then rebalancing should clear it.

1. In Configuration Manager, go to the HDFS service.

2. Select Actions and then select Rebalance.

3. Click Rebalance to continue.

Note:

Depending on data volume, rebalancing may take many hours.

If Oracle Big Data Connectors is enabled, then ORAAH may not work after the upgrade

The solution is to disable and then re-enable Oracle Big Data Connectors after the upgrade. You will need to authenticate with the Configuration Manager admin password and the MySQL root password for both operations.

# bdacli disable bdc
# bdcali enable bdc

When you re-enable Big Data Connectors, note that Big Data Appliance 5.2 does not support the ODI agent. At the following prompt, enter n :

Do you wish to enable ODI? [y/n]: n

Errors That You May Encounter During Node Reimaging

• bdareimagenode: Preparing internal USB disk failed

  If you see this error, do the following:

  1. Reboot the node.
  2. Run lsscsi to check that the disks are in the correct order.
  3. Run mount -l to verify the mounted HDFS partitions. Check for anomalies.
  4. Repeat the reimaging of the node:

     # ./bdareimagenode.sh | tee /path_to_log/bdareimagenode.log

• cp: writing `/tmp/bdausbmnt/boot/upgrade.img': No space left on device

  When re-imaging a server with a 4 GB USB drive, this error is recorded in the log file bdareimagenode-prepare-usb.out. This expected error is harmless and you can ignore it. The file upgrade.img and all files after it are not needed by the reimaging process.

10.8.2 Prerequisites for Migrating to Oracle Linux 7

Be sure that these prerequisites are met before proceeding with the migration.

• The Cluster Software Level Must be Oracle Linux 6 on Oracle Big Data Appliance 4.13.0 or Greater

• Disable or Uninstall Oracle Big Data SQL

  If Oracle Big Data SQL is installed, you must uninstall the Hadoop side of the installation before performing the OS upgrade on any nodes. It is not necessary to uninstall the database server side of the installation. Although the database side can remain intact during the OS upgrade on the Hadoop cluster, queries against data in Hadoop will not work while the upgrade is in progress and until you re-enable Big Data SQL in the cluster.

  • For Oracle Big Data SQL 3.2 and later, use the following bdacli command to disable this Oracle Big Data SQL across the cluster:

    # bdacli disable big_data_sql

  • If an earlier version of Oracle Big Data SQL is installed use the setup-bds command with the uninstall parameter and the same version of the configuration file that was use to perform the installation:

    # ./setup-bds uninstall bds-config.json

  Tip:

  If you are not sure which method to use, you can safely try bdacli disable big_data_sql. If this command fails, then it is likely that the software was installed using ./setup-bds .

  If you have not removed Oracle Big Data SQL, before the OS upgrade, the process displays a message stating that you must remove Oracle Big Data SQL and then terminates with an error.

  Reinstall Big Data SQL on the Hadoop cluster and reconnect with the database side of the installation after the upgrade. If there are connectivity problems after reinstalling Oracle Big Data SQL on the Hadoop cluster, you can try reconfiguring the installation on the database server to bring the two sides back into alignment. See ./bds-database-install.sh --reconfigure in the Oracle Big Data SQL Installation Guide.

• Uninstall eCryptfs if it is in use

  Early versions of Oracle Big Data supported the use eCryptfs, but support for this encryption method was deprecated in later versions. Upgrade of nodes that use eCryptfs encryption is not supported.

  If you have not removed eCryptfs, the OS upgrade displays a message stating that you must remove them and then terminates with an error

  You can re-encrypt the data with HDFS Transparent Encryption before or after the migration.

  HDFS Transparent Encryption and HTTPS / Network Encryption are not affected by the OS upgrade.

10.8.3 Oracle Linux 6 to Oracle Linux 7 Migration Steps

This is the procedure for upgrading a cluster from Oracle Linux 6 to Oracle Linux 7.

If you have not already done so, review previous sections in this chapter:

• Important Considerations
• Prerequisites for Migrating to Oracle Linux 7

Pay special attention to requirement to preserve critical information on another node prior to the reboot step.

1. As root, run mammoth -c to check cluster health. Resolve issues that might interfere with the operating system upgrade before proceeding further.
2. On the "Mammoth node" (where Mammoth runs, usually the first node of the cluster), download the Mammoth OL7 bundle that matches the release of Big Data Appliance currently running on OL6 to /tmp and unzip it. For example, if you are running Big Data Appliance 5.2, download and unzip Patch 30459890: ORACLE BIG DATA APPLIANCE MAMMOTH DEPLOYMENT BUNDLE V5.1.0 FOR ORACLE LINUX 7 .
3. Change to the BDAMammoth directory and run BDAMammoth*.run the with osupgrade flag. For example:
   # ./BDAMammoth-CDH-ol7-5.2.0.run -osupgrade
4. On each node of the cluster, perform the following steps. Work through the nodes one at a time. Start with the non-critical DataNodes, generally Node5 and higher. End with the Mammoth node (usually Node1). This is the most significant critical node.
   1. On the node, remove the Hadoop roles that are part of any services not managed by Mammoth. This includes, Sqoop gateway roles.

      Roles managed by Mammoth are HDFS, Impala, Sentry (as long as bdacli getinfo cluster_sentry_enabled returns true), Spark, Spark2, YARN, Hive, HUE, Oozie, and Zookeeper.” Mammoth also manages Management Service roles. All of these should remain, with the exception of any extra Management Service roles that were added manually. These Management Service roles should be removed.

      TIP: In Cloudera Manager go to Hosts>Roles. Hover over the role icons to see the role name.
      Description of the illustration cm_role_list.png
      Temporarily delete roles the roles not managed by Mammoth. It is not sufficient to stop these roles. You can restore them after the upgrade.
   2. Download patch 28218768, which contains the Oracle Big Data Appliance base image. Unzip the archive into /root.
   3. Review all of the files in /root/<BDABaseImageVersion>/osupgrade_filelist. These are the files and directories that the upgrade has predetermined should be saved. Create your own file list at /opt/oracle/BDAMammoth/bdaconfig/tmp/my_filelist if you want to preserve more files or directories.
   4. Change to the extracted BDABaseImage directory. From there, run ./bdadumpnode.sh.
   5. In the output of bdadumpnode.sh, find the backup directory path that will be used. You can see it immediately after the message, "Backup directory selected". Note down the backup directory. You will need it later in these steps.
   6. From the extracted BDABaseImage directory, run # ./bdareimagenode.sh. Again, save the output to a log.
      # ./bdareimagenode.sh | tee /path_to_log/bdareimagenode.log
      Also check /tmp/bdareimagenode-prepare-usb.out to verify that the script ran without errors.
   7. Do the following to preserve critical information before the reboot in the next step:

      Caution:

      To reduce the risk of data loss, you are strongly recommended to take these steps.

      1. Check to ensure that the label DO_NOT_WIPE is set on /dev/sda4. If it is not, set it with this command.

      e2label /dev/sda4  DO_NOT_WIPE

      2. Save the following information to a location within HDFS on a different node.

      Tip:

      It is best to store this information in HDFS on another Hadoop node, since the default three-factor replication will automatically copy it to two other nodes as well. (If you store the information outside of HDFS, then you won't get the extra protection provided by this replication.)

      • A backup copy of the entire /u*/osupgrade_backup directory after bdadumpnode.sh has completed.

        Do this in addition to including the same directory in the local /opt/oracle/BDAMammoth/bdaconfig/tmp/my_filelist. Taking both steps provides extra protection. All directories not in saved in osupgrade_filelist or my_filelist will be deleted and not preserved

      • The logs of the bdadumpnode.sh and bdareimagenode.sh output in the previous steps.
      • Backups of the output of blkid, df -h, and the lsblk command shown below:

        # blk > <nodename>_blk.log

        # df -h > <nodename>_df-h.log

        # lsblk --output NAME,MOUNTPOINT,LABEL,PARTLABEL > <nodename>_lsblk.log 

   8. After saving critical information, reboot the server:
      # reboot
   9. After the node has finished reimaging and is back online, log in and run uname -r to confirm that the OS is now at Oracle Linux 7.
      Then, check for /root/BDA_IMAGING_*. If this directory exists, then reimaging has succeeded.
   10. If there are no indications of errors, then reboot the server again.
       After the reboot is complete, check for the existence of /root/BDA_REBOOT_* to confirm that the reboot succeeded. (It is normal for these files to be empty.)
   11. After reimaging is complete and the server has rebooted, execute this script:
       # /root/Extras/bdarestorenode.sh
       Ensure that this script completes successfully.
   12. Optional Step: Check After Oracle Linux 6 to Oracle Linux 7 Migration on BDA 5.1.0, the Oracle Linux OL7 RPMs are Not the Same as on a New or Upgraded BDA v5.1.0 Cluster (Doc ID 2677625.1) for optional RPM updates.
   13. Perform any server-specific customizations or third-party software installations needed on this node.
   14. Log on to the next node. Repeat the process on all non-critical nodes, and then on the critical nodes. Leave the "Mammoth node" until last. This is the node where Mammoth runs, usually Node1.
5. Log on to the Mammoth node (again, usually Node1) and complete the same steps above. Be sure to run /root/Extras/bdarestorenode.sh as the last step.
6. Do a final check of cluster health:
   # mammoth -c

10.8.4 Output Examples From Oracle Linux 6 to Oracle Linux 7 Migration

The following examples are provided to show you expected output from operations performed during the migration to Oracle Linux 7. There may be some differences between these examples and the output you observe on your own systems.

bdadumpnode.sh

# ./bdadumpnode.sh
bdadumpnode: working directory         : /root/BDABaseImage-ol7-4.10.0_190314
Enter CM admin password:
Enter CM admin password again:

bdadumpnode: CM REST API version       : v19
Enter MySQL root password:
Enter MySQL root password again:

bdadumpnode: required disk space(1K blocks) : 18163
bdadumpnode: Backup directory selected : /u01/osupgrade_backup
bdadumpnode: HDFS Transparent Encryption is enabled on this cluster
bdadumpnode: Stopping all hadoop roles on host : <node name>13
bdadumpnode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/cm_commands.out
bdadumpnode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/hosts_7634f839-f511-44d7-b2dd-a9991230291b_view_full_1552690645.out
bdadumpnode: Stopping yarn - yarn-NODEMANAGER-2bb159eb8269eec214d88ac80364bf9a
bdadumpnode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_services_yarn_roleCommands_stop_1552690646.out
bdadumpnode: Command 1613 finished after 5 seconds
bdadumpnode: Operation completed successfully
bdadumpnode: Stopping hdfs - hdfs-DATANODE-2bb159eb8269eec214d88ac80364bf9a
bdadumpnode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_services_hdfs_roleCommands_stop_1552690654.out
bdadumpnode: Command 1616 finished after 5 seconds
bdadumpnode: Operation completed successfully
bdadumpnode: Stopping Cloudera Manager Agent and Server on host : <node name>13
Stopping cloudera-scm-agent: [60G[[0;32m  OK  [0;39m]
bdadumpnode: Backing up files on host  : <node name>13
bdadumpnode: Copying                   : /etc/auto_direct_bda
bdadumpnode: Copying                   : /etc/auto.master
bdadumpnode: Copying                   : /etc/cloudera-scm-agent
bdadumpnode: Copying                   : /etc/cloudera-scm-server
bdadumpnode: Copying                   : /etc/dnsmasq.conf
bdadumpnode: Copying                   : /etc/exports
bdadumpnode: Copying                   : /etc/group
bdadumpnode: Copying                   : /etc/hosts
bdadumpnode: Copying                   : /etc/my.cnf
bdadumpnode: Copying                   : /etc/passwd
bdadumpnode: Copying                   : /etc/resolv.conf
bdadumpnode: Copying                   : /etc/shadow
bdadumpnode: Copying                   : /etc/ssh
bdadumpnode: Copying                   : /etc/yum.conf
bdadumpnode: Copying                   : /etc/yum.repos.d/bda.repo
bdadumpnode: Copying                   : /home
bdadumpnode: Copying                   : /opt/cloudera/csd
bdadumpnode: Copying                   : /opt/cloudera/security
bdadumpnode: Copying                   : /opt/exportdir
bdadumpnode: Copying                   : /opt/hadoop
bdadumpnode: Copying                   : /opt/oracle/bda/bin/krb5prop.sh
bdadumpnode: Copying                   : /opt/oracle/bda/cluster-hosts-infiniband
bdadumpnode: Copying                   : /opt/oracle/bda/.imagehistory
bdadumpnode: Copying                   : /opt/oracle/bda/install/log
bdadumpnode: Copying                   : /opt/oracle/bda/install/state
bdadumpnode: Copying                   : /opt/oracle.cellos
bdadumpnode: Copying                   : /opt/oracle.ExaWatcher
bdadumpnode: Copying                   : /opt/oracle/odiagent
bdadumpnode: Copying                   : /opt/oracle.oswatcher
bdadumpnode: Copying                   : /opt/oracle/oxh-4.9.1-1.cdh5.0.0
bdadumpnode: Copying                   : /root/.ssh
bdadumpnode: Copying                   : /usr/java/default/jre/lib/security/local_policy.jar
bdadumpnode: Copying                   : /usr/share/java/mysql-connector-java-commercial-5.1.34-bin.jar
bdadumpnode: Copying                   : /usr/share/java/mysql-connector-java.jar
bdadumpnode: Copying                   : /var/lib/cloudera-scm-agent
bdadumpnode: Copying                   : /var/lib/cloudera-scm-navigator
bdadumpnode: Copying                   : /var/lib/cloudera-scm-server
bdadumpnode: Copying                   : /var/lib/oozie
bdadumpnode: Copying                   : /var/lib/zookeeper
****************************************************
bdadumpnode: Node dump completed successfully on host : <node name>13
****************************************************

bdareimagenode.sh

# ./bdareimagenode.sh
bdareimagenode: Using BDABaseImage-ol7-4.10.0_190314.iso to reimage this node
BDABaseImage-ol7-4.10.0_190314.iso: OK
bdareimagenode: Using node nr             : 13
bdareimagenode: Using network files: /opt/oracle/bda/rack-network.json & /opt/oracle/bda/cluster-network.json
bdareimagenode: Running makebdaimage...
bdareimagenode: Resetting disk labels of data partitions to DO_NOT_WIPE on specified servers
hba slot: s0p4
hba slot: s1p4
hba slot: s2p1
hba slot: s3p1
hba slot: s4p1
hba slot: s5p1
hba slot: s6p1
hba slot: s7p1
hba slot: s8p1
hba slot: s9p1
hba slot: s10p1
hba slot: s11p1
bdareimagenode: Internal USB disk has been prepared. All disk labels have been set to DO_NOT_WIPE
bdareimagenode: Please check log /tmp/bdareimagenode-prepare-usb.out. If no error then reboot this server

bdarestorenode.sh

# ./bdarestorenode.sh
bdarestorenode: working directory         : /root
Enter CM admin password:
Enter CM admin password again:

bdarestorenode: CM REST API version       : v19
bdarestorenode: restoring systhem data on host : <node name>13
bdarestorenode: working directory         : /u01/osupgrade_backup
bdarestorenode: merging passwd, shadow and group
bdarestorenode: restoring /etc directories
bdarestorenode: restoring /var/lib directories
bdarestorenode: restoring /opt directories
bdarestorenode: restoring ssh, jce, and mysql java directories
bdarestorenode: adding domain to /etc/idmapd.conf
Loaded plugins: langpacks
=============================== repo: ol7_UEKR4 ================================
[ol7_UEKR4]
async = True
bandwidth = 0
base_persistdir = /var/lib/yum/repos/x86_64/7Server
baseurl = https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/
cache = 0
cachedir = /var/cache/yum/x86_64/7Server/ol7_UEKR4
check_config_file_age = True
compare_providers_priority = 80
cost = 1000
deltarpm_metadata_percentage = 100
deltarpm_percentage = 
enabled = 0
enablegroups = True
exclude = 
failovermethod = priority
ftp_disable_epsv = False
gpgcadir = /var/lib/yum/repos/x86_64/7Server/ol7_UEKR4/gpgcadir
gpgcakey = 
gpgcheck = True
gpgdir = /var/lib/yum/repos/x86_64/7Server/ol7_UEKR4/gpgdir
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
hdrdir = /var/cache/yum/x86_64/7Server/ol7_UEKR4/headers
http_caching = all
includepkgs = 
ip_resolve = 
keepalive = True
keepcache = False
mddownloadpolicy = sqlite
mdpolicy = group:small
mediaid = 
metadata_expire = 21600
metadata_expire_filter = read-only:present
metalink = 
minrate = 0
mirrorlist = 
mirrorlist_expire = 86400
name = Latest Unbreakable Enterprise Kernel Release 4 for Oracle Linux 7Server (x86_64)
old_base_cache_dir = 
password = 
persistdir = /var/lib/yum/repos/x86_64/7Server/ol7_UEKR4
pkgdir = /var/cache/yum/x86_64/7Server/ol7_UEKR4/packages
proxy = False
proxy_dict = 
proxy_password = 
proxy_username = 
repo_gpgcheck = False
retries = 10
skip_if_unavailable = False
ssl_check_cert_permissions = True
sslcacert = 
sslclientcert = 
sslclientkey = 
sslverify = True
throttle = 0
timeout = 30.0
ui_id = ol7_UEKR4/x86_64
ui_repoid_vars = releasever,
   basearch
username = 

=============================== repo: ol7_latest ===============================
[ol7_latest]
async = True
bandwidth = 0
base_persistdir = /var/lib/yum/repos/x86_64/7Server
baseurl = https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/
cache = 0
cachedir = /var/cache/yum/x86_64/7Server/ol7_latest
check_config_file_age = True
compare_providers_priority = 80
cost = 1000
deltarpm_metadata_percentage = 100
deltarpm_percentage = 
enabled = 0
enablegroups = True
exclude = 
failovermethod = priority
ftp_disable_epsv = False
gpgcadir = /var/lib/yum/repos/x86_64/7Server/ol7_latest/gpgcadir
gpgcakey = 
gpgcheck = True
gpgdir = /var/lib/yum/repos/x86_64/7Server/ol7_latest/gpgdir
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
hdrdir = /var/cache/yum/x86_64/7Server/ol7_latest/headers
http_caching = all
includepkgs = 
ip_resolve = 
keepalive = True
keepcache = False
mddownloadpolicy = sqlite
mdpolicy = group:small
mediaid = 
metadata_expire = 21600
metadata_expire_filter = read-only:present
metalink = 
minrate = 0
mirrorlist = 
mirrorlist_expire = 86400
name = Oracle Linux 7Server Latest (x86_64)
old_base_cache_dir = 
password = 
persistdir = /var/lib/yum/repos/x86_64/7Server/ol7_latest
pkgdir = /var/cache/yum/x86_64/7Server/ol7_latest/packages
proxy = False
proxy_dict = 
proxy_password = 
proxy_username = 
repo_gpgcheck = False
retries = 10
skip_if_unavailable = False
ssl_check_cert_permissions = True
sslcacert = 
sslclientcert = 
sslclientkey = 
sslverify = True
throttle = 0
timeout = 30.0
ui_id = ol7_latest/x86_64
ui_repoid_vars = releasever,
   basearch
username = 

Unpacking JAR files...
	tools.jar...
	plugin.jar...
	javaws.jar...
	deploy.jar...
	rt.jar...
	jsse.jar...
	charsets.jar...
	localedata.jar...
bdarestorenode: checking ol7 bundle directory on node : <node name>09
bdarestorenode: Ensure CDH ol7 parcel in the remote temp repo
bdarestorenode: Ensure Cloudera Manager ol7 pkgs in the bdarepo
bdarestorenode: Ensure MySQL ol7 pkgs in the bdarepo
Loaded plugins: langpacks, ulninfo
Cleaning repos: bda
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: aliases, changelog, filter-data, keys, list-data, ps, security,
              : verify
Cleaning repos: bda
Cleaning up Everything
Could not find valid repo at: /var/www/html/bda
Spawning worker 0 with 1528 pkgs
Workers Finished
Gathering worker results

Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
bdarestorenode: restoring Kerberos directories
bdarestorenode: installing MySQL ol7 pkgs on host : <node name>13
bdarestorenode: restoring my.cnf
bdarestorenode: installing cloudera manager pkgs
bdarestorenode: restoring cloudera directories
bdarestorenode: starting cloudera manager services
Starting cloudera-scm-agent (via systemctl):  [60G[[0;32m  OK  [0;39m]
bdarestorenode: waiting for CDH parcel to automatically deploy on node : <node name>13
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/cm_commands.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/cm_config_1552708777.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_CDH_versions_5.15.1-1.cdh5.15.1.p0.4_1552708957.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_CDH_versions_5.15.1-1.cdh5.15.1.p0.4_1552709018.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_CDH_versions_5.15.1-1.cdh5.15.1.p0.4_1552709079.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_CDH_versions_5.15.1-1.cdh5.15.1.p0.4_1552709139.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_CDH_versions_5.15.1-1.cdh5.15.1.p0.4_1552709200.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_CDH_versions_5.15.1-1.cdh5.15.1.p0.4_1552709261.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_CDH_versions_5.15.1-1.cdh5.15.1.p0.4_1552709322.out
bdarestorenode: CDH parcel activated successfully! : CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel
bdarestorenode: waiting for SPARK2 parcel to automatically deploy on node : <node name>13
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_SPARK2_versions_2.3.0.cloudera3-1.cdh5.13.3.p0.458809_1552709325.out
bdarestorenode: Spark2 parcel activated successfully! : SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809-el7.parcel
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_KEYTRUSTEE_SERVER_versions_5.15.0-1.keytrustee5.15.0.p0.278282_1552709329.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_KEYTRUSTEE_SERVER_versions_5.15.0-1.keytrustee5.15.0.p0.278282_1552709389.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_KEYTRUSTEE_SERVER_versions_5.15.0-1.keytrustee5.15.0.p0.278282_1552709450.out
bdarestorenode: Key Trustee Server parcel activated successfully! : 5.15.0-1.keytrustee5.15.0.p0.278282
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_scajvm_parcels_products_KEYTRUSTEE_versions_5.15.1-1.KEYTRUSTEE5.15.1.p0.484462_1552709453.out
bdarestorenode: KeyTrustee parcel activated successfully! : KEYTRUSTEE-5.15.1-1.KEYTRUSTEE5.15.1.p0.484462-el7.parcel
bdarestorenode: start_hadoop_roles_in_current_host
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/cm_commands.out
bdarestorenode: Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/cm_commands_hostsStartRoles.out
.......bdarestorenode: Command 1621 finished after 40 seconds
bdarestorenode: Operation completed successfully
bdarestorenode: Installing OSG
bdarestorenode: Installing Oracle R and ORAAH : <node name>13
bdarestorenode: Installing Oraloader
bdarestorenode: Installing Oxh
bdarestorenode: /u01/osupgrade_backup/my_filelist not found. No customer data to restore
*****************************************************
bdarestorenode: OS upgrade completed successfully on host : <node name>13
*****************************************************

10.9 Enabling Optional Software Included With the Release

Oracle Big Data Appliance includes a number of optional software packages. If you did not choose to install these packages in the initial installation, you can reverse some of those decisions later. In some cases, you can quickly enable optional software using the bdacli utility. In other cases, more setup and configuration steps are necessary. Be prepared to provide relevant server names, ports, user names, and passwords where required.

This section provides examples of some reconfiguration options:

• Enable or Disable Oracle Big Data Connectors

• Enable or Disable Auto Service Request

• Add Support for Oracle Enterprise Manager Cloud Control

• Enable Kerberos Authentication

• Installing Oracle Big Data SQL

10.9.1 Enable or Disable Oracle Big Data Connectors

Oracle Big Data Connectors is included in the Oracle Big Data Appliance deployment bundle. Oracle Big Data Appliance customers do not need to download Oracle Big Data Connectors from an external source. However, you can download and install a newer version if it has been approved for use on this release of Oracle Big Data Appliance.

Oracle Big Data Connectors includes the following:

• Oracle Loader for Hadoop

• Oracle SQL Connector for Hadoop Distributed File System

• Oracle XQuery for Hadoop

• Oracle R Advanced Analytics for Hadoop

• Oracle Datasource for Apache Hadoop

Prior to the Oracle Big Data Appliance software installation, you can use the Oracle Big Data Appliance Configuration Generation Utility to instruct Mammoth to enable Oracle Big Data Connectors at installation time. At any time after the installation you can use the bdacli utility to enable or disable Oracle Big Data Connectors:

# bdacli [enable|disable] bdc

See Also:

• Using Oracle Big Data Appliance Configuration Generation Utility

• bdacli

• Oracle Big Data Connectors User’s Guide

The following sections provide more detail on enabling and disabling Oracle Big Data Connectors.

10.9.1.1 Adding Oracle Big Data Connectors

Note:

Oracle Big Data Appliance 5.1 does not support the ODI (Oracle Data Integrator) agent, therefore you do not have the option to install the agent in this release.

The following procedure uses the bdacli utility.

To add Oracle Big Data Connectors to a cluster:

1. Log in to the first NameNode (Node1) of the primary rack as root.

2. Enable Oracle Big Data Connectors:

   # bdacli enable bdc
   INFO: Logging all actions in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node03-20140805110007.log and traces in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node03-20140805110007.trc
   INFO: This is the install of the primary rack
   INFO: Checking if password-less ssh is set up
        .
        .
        .
   Enter password for the mysql root user
   Enter password: root_password
   Enter password again: root_password
   WARNING: The password for the Cloudera Manager admin user is missing from the parameters file and is required for the installation.
   Enter password: admin_password
   Enter password again: admin_password
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     2    0     2    0     0    181      0 --:--:-- --:--:-- --:--:--   250
        .
        .
        .
   SUCCESS: Successfully reconfigured service

10.9.1.2 Removing Oracle Big Data Connectors

When removing support for Oracle Big Data Connectors, you must provide the password for the MySQL Database root user if the password is not saved in cluster_name-config.json:

The following procedure uses the bdacli utility.

To remove Oracle Big Data Connectors from a cluster:

1. Log in to the first NameNode (node01) of the primary rack as root.

2. Remove Oracle Big Data Connectors:

   # bdacli disable bdc
   INFO: Logging all actions in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node03-20140805104603.log and traces in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node03-20140805104603.trc
   INFO: This is the install of the primary rack
   INFO: Checking if password-less ssh is set up
   INFO: Executing checkRoot.sh on nodes 
    
   /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   SUCCESS: Executed checkRoot.sh on nodes 
    
   /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   INFO: Executing checkSSHAllNodes.sh on nodes 
    
   /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   SUCCESS: Executed checkSSHAllNodes.sh on nodes 
    
   /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   INFO: Reading component versions from 
    
   /opt/oracle/BDAMammoth/bdaconfig/COMPONENTS
   INFO: Creating nodelist files...
   WARNING: The password for the Cloudera Manager admin user is missing from the parameters file and is required for the installation.
   Enter password: admin_password
   Enter password again: admin_password
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     2    0     2    0     0    184      0 --:--:-- --:--:-- --:--:--   250
   WARNING: The password for the MySQL root user is missing from the parameters file and is required for the installation.
   Enter password: root_password
   Enter password again: root_password
   INFO: Executing verifyMySQLPasswd.sh on nodes 
   /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   SUCCESS: Executed verifyMySQLPasswd.sh on nodes 
        .
        .
        .   
   /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   INFO: Removing big data connectors. This will take some time ...
        .
        .
        .
   SUCCESS: Successfully reconfigured service

10.9.2 Enable or Disable Oracle Big Data Spatial and Graph

Oracle Big Data Spatial and Graph is included in the Oracle Big Data Appliance deployment bundle. Oracle Big Data Appliance customers do not need to download Oracle Big Data Spatial and Graph from an external source.

You can enable or disable this software at any time. These are the options:

• Prior to the Oracle Big Data Appliance software installation, you can use the Oracle Big Data Appliance Configuration Generation Utility to instruct Mammoth to enable Oracle Big Data Spatial and Graph at installation time.

• At any time after the installation you can use the bdacli utility to enable or disable Oracle Big Data Spatial and Graph.

# bdacli [enable|disable] osg

See Also:

• Using Oracle Big Data Appliance Configuration Generation Utility

• The bdacli reference in this guide.

10.9.3 Enable or Disable Auto Service Request

The following procedure shows how to add support for Auto Service Request.

To support Auto Service Request:

1. Set up your My Oracle Support account and install ASR Manager. You must do this before activating Auto Service Request on Oracle Big Data Appliance. See Setting Up Auto Service Request.

2. Turn on Auto Service Request monitoring and activate the assets:

   # bdacli enable asr
   INFO: Logging all actions in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node01-20130205075303.log and traces in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node01-20130205075303.trc
        .
        .
        .
   Enter the value for ASR_HOST [Default: ]: asr-host.example.com
   Enter the value for ASR_PORT [Default: 162]:
   Enter the value for ASR_SERVER_USER: jdoe
    
   Please Check the values you entered for the ASR parameters
    
   ASR_HOST = asr-host.example.com
   ASR_PORT = 162
   ASR_SERVER_USER = jdoe
    
   Are these values correct (y/n): y
   Enter password for user jdoe on machine asr-host.example.com
   Enter password: password
   Enter password again: password
   INFO: Setting up ASR on all nodes. This will take some time ...
        .
        .
        .
   

3. Complete the steps in "Verifying the Auto Service Request Installation."

10.9.4 Enable Kerberos Authentication

The following procedure configures Kerberos authentication.

Note:

If you choose to enable Active Directory Kerberos, first read MOS (My Oracle Support) documents 2029378.1 and 2013585.1. These documents explain required preliminary steps and provide important information on known issues.

To support Kerberos authentication:

1. Ensure that you complete the Kerberos prerequisites listed in "Installation Prerequisites."

2. Log into the first NameNode (Node1) of the primary rack.

3. Configure Kerberos:

   # bdacli enable kerberos
   INFO: Logging all actions in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node01-20131104072502.log and traces in /opt/oracle/BDAMammoth/bdaconfig/tmp/bda1node01-20131104072502.trc
        .
        .
        .
   INFO: Executing checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   SUCCESS: Executed checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
   SUCCESS: Password-less root SSH is setup.
    
    Do you want to setup KDC on a BDA node (yes/no): yes
    
    Please enter the realm name: EXAMPLE.COM
    
    Enter password for Kerberos database: password
    Enter password again: password
   INFO: Executing changekrb5_kdc.sh on nodes bda1node01 #Step -1#
   SUCCESS: Executed changekrb5_kdc.sh on nodes bda1node01 #Step -1#
   SUCCESS: Successfully set the Kerberos configuration on the KDC
   INFO: Setting up Master KDC
   INFO: Executing setupKDC.sh on nodes bda1node01 #Step -1#
        .
        .
        .

Services Automatically Configured for Kerberos by bdacli

Using bdacli enable kerberos configures Kerberos for the cluster as a whole and also for services that are pre-configured by the Mammoth installer: HDFS, YARN, Spark, Hive, Hue, Oozie, and Zookeeper. For these services, no additional steps are needed.

For services that you have added and configured such as HBase or Flume, review the Cloudera documentation on using Cloudera Manager to configure Kerberos for each service.

10.9.5 Installing Oracle Big Data SQL

Oracle Big Data SQL supports queries against non-relational data stored in multiple big data sources, including Apache Hive, HDFS, Oracle NoSQL Database, Apache HBase, and other NoSQL databases, as well as Oracle Cloud Infrastructure Object Storage, Amazon S3, and Microsoft Azure.

A license is required to install Oracle Big Data SQL on Oracle Big Data Appliance. No additional license is required for the Oracle Database side of the installation.

Big Data SQL components must be installed on both Hadoop and Oracle Database. The Oracle Big Data SQL Installation Guide provides instructions for both parts of the installation, including generic instructions for installing the product on several supported Hadoop frameworks. However, on Oracle Big Data Appliance, most of the Hadoop-side installation is integrated with Mammoth, the Big Data Appliance installer. Installing Oracle Big Data SQL in this guide explains what is different about the Hadoop-side installation of Big Data SQL on Big Data Appliance.

Supported Connections to Oracle Database

When installed on Oracle Big Data Appliance, Oracle Big Data SQL can be configured to connect the following Oracle Database systems:

• Oracle Exadata Database Machine

• Oracle Database on Oracle Linux or RHEL-based commodity servers. Specific systems are listed in the Oracle Big Data SQL Master Compatibility Matrix (Doc ID 2119369.1 in My Oracle Support)

Oracle Big Data SQL can connect to both single-instance Oracle Database and multinode databases (such as Oracle RAC).

See Also:

Oracle Big Data SQL Documentation.

Installing Over an Existing Oracle Big Data SQL Installation

It is not necessary to uninstall a previous version of the software. The installation will upgrade the previously installed version to the current release level.

Note:

If a previous of Oracle Big Data SQL already exists on the Oracle Big Data Appliance, the Mammoth installer will upgrade this installation to the current release level regardless of whether or not you select Oracle Big Data SQL for installation via the form fields in the Configuration Generation Utility.

On Oracle Big Data Appliance, the software is installed at /opt/oracle/BDSJaguar on the configuration management server (the node where Cloudera Configuration Manager is running).

Automatic Setup of Ticket Renewal for Kerberos-Secured Environments

If your environment is Kerberos-secured, the installation automatically sets up a cron job to renew the ticket for the oracle account principal four times per day. This frequency is to satisfy the AD Kerberos requirement. The extra kinits are harmless for MIT Kerberos. The automation is set up as needed for both Oracle Big Data Appliance and the Oracle Database system.

10.9.5.1 Installation Options

Each Oracle Big Data Appliance software release already includes a version of Oracle Big Data SQL that is ready to install, using the utilities available on the appliance.

You have the option of using Big Data Appliance utilities as described in this guide. The other option is to download and install a "standalone" release of Big Data SQL from the Oracle Software Delivery Cloud (edelivery.oracle.com) if a newer version that is compatible with this Big Data Appliance release is available. But for Big Data Appliance, the recommended method is to install the Big Data SQL package included with your Big Data Appliance software. .

The advantages of installing the version of Big Data SQL included with the appliance are:

• The software prerequisites to the installation are already met.
• You can add Big Data SQL to the Big Data Appliance release installation by checking a checkbox in the Big Data Appliance Configuration Generation Utillity. The Mammoth utility will then automatically include Big Data SQL in the installation.
• You can also install Big Data SQL later, using the bdacli utility. This is also a simple procedure. The command is bdacli enable big_data_sql.
• During the upgrade to a newer Big Data Appliance software release, Mammoth will automatically upgrade the Hadoop side of the Big Data SQL installation to the version included in the release bundle if the version included in the Big Data Appliance release bundle is newer. It does this whether the version of Big Data SQL already on the appliance was installed by Mammoth itself or from the standalone installation bundle.

The limitations of installing the version of Big Data SQL include with Big Data Appliance are:

• The installation is performed for the Hadoop side only. You still need to install the database side of the product using the instructions in this guide. You also must refer to this guide if you want to modify the default installation.
• The Big Data Appliance release may not include the latest available version of Big Data SQL.

This guide explains how to install Big Data SQL 4.1 using the Mammoth or bdacli utilities. The alternate method is to download Release 4.1 from the Oracle Software Delivery Cloud and install it using its own "standalone" installer. This method is described in the Oracle Big Data SQL Installation Guide.

10.9.5.2 Installation Overview

Oracle Big Data SQL includes an installation on Oracle Big Data Appliance and another installation on the database system that will query data on the appliance.

Phases of the Installation

1. Installing on Oracle Big Data Appliance.

   The Oracle Big Data Appliance Configuration Generation Utility provides form fields to direct Mammoth to include Oracle Big Data SQL in your installation.

   You can also use the bdacli utility enable Oracle Big Data SQL at any time after the Mammoth installation:

2. Installing on the Oracle Database system.

   You must do this part of the installation manually. A database-side installation bundle is generated when you run the Oracle Big Data Appliance installation. You must copy this bundle over to each compute node of the Oracle Database system, unpack it, and install it.

3. Enabling optional security and user access features (not part of the default installation)

   You can customize the Oracle Big Data SQL installation to include features that are not configurable in the default installation on Oracle Big Data Appliance, such as:

   • Query Server, a lightweight, zero-maintenance 18c Oracle Database that you install directly on a Hadoop cluster edge node. It gives you an easy way query data in Hadoop without the need for a full-fledged Oracle Database system. The service consists of the Oracle SQL query engine only. It provides no persistent storage except for certain categories of metadata that are useful to retain across sessions.
   • Multi-User Authorization, which enables users other than the database owner to run Oracle Big Data SQL queries against data in the Hadoop clusters on the appliance. Completing the activation of this feature (as well as Database Authentication, which is enabled by default) requires a third step in the installation process.

See Also:

Instructions on how to install the database-side bundle and enable optional features are provided in the Oracle Big Data SQL Installation Guide.

10.9.5.3 Using Mammoth or bdacli to Install the Big Data Appliance Side of Oracle Big Data SQL

Big Data SQL can be installed via the Mammoth or bdacli utilities. If Mammoth detects that an earlier releases of Big Data SQL is aleady installed, it will automatically upgrade the Big Data Appliance side of the installation. The database side must be upgraded manually.

Note:

The installation of Big Data SQL via Mammoth or bdacli does not include the Query Server feature. You can add Query Server by reconfiguring the installation later.

If This is a First-Time Installation (no Earlier Release Installed)

Directing Mammoth to install the software:

You can include Oracle Big Data SQL in the Mammoth installation by selecting it in the Oracle Big Data Appliance Configuration Generation Utility. You must make the selection for each cluster where you want to install the software.

Set up the Oracle Big Data SQL installation on Cluster Definition page of the Configuration Generation Utility as shown in the figure below.

Figure 10-1 Oracle Big Data SQL Fields in the Configuration Generation Utility

1. If you select Yes to the licensing question, you can proceed to the other options. Also select Yes in response to “Install Big Data SQL?”

2. Specify whether Oracle Big Data Appliance is connected to the Oracle Database system over an Ethernet or InfiniBand connection. Both are supported.

When you generate the configuration files, the directive to run the Oracle Big Data SQL installation is added to the <cluster name>-config.json file.

Performing a post-Mammoth installation, using bdacli:

If you did not include Oracle Big Data SQL in the Mammoth installation, you can use the bdacli utility to enable it at any time after the Mammoth installation:

# bdacli enable big_data_sql

Both Mammoth and bdacli install Oracle Big Data SQL on all of the DataNodes of the selected cluster.

Note:

In an Oracle Linux 6 cluster only, you must install the Python cryptography module on the node where Configuration Manager is hosted (usually Node3) before you can use Big Data SQL. Download pip from the Internet, install pip, and then use pip to install the cryptography module. Use scl to invoke Python 2.7 for the Python commands:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
scl enable python27 'python get-pip.py'
scl enable python27 'pip install cryptography'

Generating a GUID-Key Pair for Database Authentication

In order to make use of the Database Authentication feature described in the Oracle Big Data SQL Installation Guide, you must generate and install a GUID-key pair. This process starts by running the following command on the node where Cloudera Configuration Manager is installed in order to generate the key:

# ./jaguar --requestdb <database name> databasereq

You then copy the file that contains that key to the Oracle Database server prior to running the database-side installation of Oracle Big Data SQL. There are several other steps to complete the process of enabling Database Authentication. Read through Installing or Upgrade the Hadoop Side of Oracle Big Data SQL in the Oracle Big Data SQL Installation Guide to learn about the Jaguar utility used to generate the key file and about the complete key generation and installation process.

If an Earlier Release of Oracle Big Data SQL is Already Installed

In this case, you do not need to select Oracle Big Data SQL in the Configuration Generation Utility. Mammoth will automatically upgrade the earlier Oracle Big Data SQL release to the current version on all clusters where it is currently installed. However, this automatic upgrade is performed on the Hadoop side only. You must manually upgrade the database side of Oracle Big Data SQL as described in this guide.

10.9.5.4 Installing Oracle Big Data SQL on the Oracle Database System

When you install Oracle Big Data SQL on Oracle Big Data Appliance, this process also generates a database-side installation bundle. Install this bundle on the database system where you intend to run Oracle Big Data SQL queries against the Hadoop data on Oracle Big Data Appliance.

Installing the database side of Oracle Big Data SQL is not covered in this guide. For instructions, refer to Installing or Upgrade Oracle Big Data SQL on the Oracle Database System in the Oracle Big Data SQL Installation Guide.

Note:

Before you start, ensure that the installer has access to the RPMs for clients the Big Data SQL needs in order to access the Hadoop environment from the Oracle Database system.

10.9.5.4.1 Giving the Installer Access to the Repository Where Required Client RPMs are Stored

In order to communicate with the Hadoop environment in Oracle Big Data Appliance 5.2, the Big Data SQL installation on the Oracle Database server requires the CDH 6.3.4 version of the Hadoop and Hive clients. The HBase client is also required if you intend to use HBase. You must provide the installer access to client RPMs. These are your options:

• Configure the Big Data SQL installer to download the client RPMs directly from the Cloudera archive
• Download the client RPMs to a local repository first and then configure the installer to pull the client RPMs from the local repository.

Passing Credentials to Authenticate Against the Cloudera Repository

Both of the options described above require credentials from Cloudera as described in the Cloudera documentation at Authentication to the Cloudera Archive

1. Go to Authentication to the Cloudera Archive in the Cloudera documentation. Follow the instructions provided to obtain your Cloudera credentials.
2. When you run bds-database-install.sh, the database-side installer, include the --alternate-repo parameter and provide you credentials as described in the topic Command Line Parameter Reference for bds-database-install.sh in the Oracle Big Data SQL 4.1 Installation Guide.

10.9.5.5 Customizing the Default Oracle Big Data SQL Installation

When you use Mammoth or bdacli to install Big Data SQL, these utilities perform a basic installation that does not enable all of the features that are available.

After the basic installation, you can use the Jaguar utility that comes with Big Data SQL to enable additional features and make other changes. For example:

• Add Multi-User Authorization.

  In previous releases of Oracle Big Data SQL, all queries against Hadoop and Hive data are executed as the oracle user and there is no option to change users. Although oracle is still the underlying user in all cases, Oracle Big Data SQL can now use Hadoop Secure Impersonation to direct the oracle account to execute tasks on behalf of other designated users. This enables HDFS data access based on the user that is currently executing the query, rather than the singular oracle user.

• Add Database Authentication.

  This is authentication of Oracle Database connections to Oracle Big Data SQL processes in the Hadoop cluster. It provides a safeguard against impersonation attacks, in which a rogue service attempts to establish a connection.

• Add Query Server.

  You can install Query Server on an cluster-managed or unmanaged edge node.

• Switch the network used by Oracle Big Data SQL.

  If the physical connections to support it are in place, you can switch Oracle Big Data Appliance and Oracle Database server from Ethernet to InfiniBand or vice versa without the need to reinstall Oracle Big Data SQL.

You can reconfigure the installation again later to reverse any of these changes.

Using the Jaguar Utility to Reconfigure the Installation

On the cluster node where Cloudera Configuration Manager is installed (usually node 3), you can find the Big Data SQL installation directory at /opt/oracle/BDSJaguar. Use the Jaguar utility in this directory to reconfigure the default installation. Jaguar reads a JSON file in which you set the configurable parameters of the installation. The default configuration file is bds-config.json, but you can use any name. The recommended method for starting a custom configuration is to make a copy of the default bds-config.json created by Mammoth and add your customizations to this file.

When you run jaguar reconfigure, the operation reads the parameters you have set in the configuration file, modifies the Oracle Big Data Appliance side of the installation accordingly, and also generates a new database-side installation bundle that supports the feature changes you have made.

Note:

Jaguar provides a number of different operations, including install and uninstall. However if you have installed the version of Oracle Big Data SQL that is bundled with Oracle Big Data Appliance, do not run the Jaguar install and uninstall operations. Use the Configuration Generation Utility or bdacli to enable or disable the software.

See Also:

About the Jaguar Utility and Jaguar Configuration Parameter and Command Reference in the Oracle Big Data SQL Installation Guide

Important:

If you create and use a custom configuration file, be sure to make a backup of the file and store it in a safe location. The file is the record of your configuration and you may need it to recreate or modify the configuration later. This is not only to protect it from accidental deletion or modification by a user. The following Mammoth and bdacli operations remove or overwrite the Oracle Big Data SQL installation directory, /opt/oracle/BDSJaguar.

• A Mammoth extension (# mammoth -e) or Mammoth upgrade (# mammoth -p) deletes the Oracle Big Data SQL installation directory.

• A reprovisioning (# bdacli admin-cluster reprovision <node name>) prompts you to delete the directory.

• A re-run of # bdacli install will overwrite your custom configuration with the default configuration.

10.9.5.5.1 Adding Query Server

You can reconfigure an existing Oracle Big Data SQL installation to include Query Server.

Query Server is an Oracle Database instance that you can optionally install as a component of Big Data SQL on an edge node in your Hadoop cluster. You use Query Server to primarily query data stored in the cluster (in HDFS and Hive formats) using Oracle external tables. This enables you to take advantage of the full SQL capabilities provided by the Oracle Database.

This section explains how to install Query Server. Working With Query Server in the Oracle Big Data SQL User's Guide explains how you can use Query Server.

10.9.5.5.2 Prequisites for Query Server

The following must be set up on Big Data Appliance before you install Query Server.

• A dedicated edge node

  Query Server must be installed on an edge node. It is highly recommended that this node is dedicated. Query Server cannot run on a node that is running the DataNode role, nor the BDSSERVER role.

  Requirements for the edge node:

• The QueryServer package from the Oracle Software Delivery Cloud (edelivery.com)

  Instructions for downloading and deploying the package are provided in this section.

• Query Server-specific configuration parameters that you set in /opt/oracle/BDSjaguar/bds-config.json
  The bds-config.json file informs the reconfigure operation which features should be enabled or disabled, provides path information, and other information needed to complete the operation. At a minimum, you must configure the parameters in the edgedb section of the configuration file. In this section, you specify that edge node is enabled for use by Query Server (you can also disable it here) and provide the address. You can also add a list of Hive databases that you want Query Server to synchronize in order to pick up updates. For example:

  "edgedb": {
  "node" : "dbnode.domain.com",
  "enabled" : "true",
  "sync_hive_db_list" : "my_hive_db_1,my_hive_db2"
  }

  If you are using Kerberos, you can set up additional Kerberos parameters to allow multiple users to access Query Server. Otherwise, only the oracle user has access. You can find details in the Jaguar Configuration Parameter and Command Reference in the Oracle Big Data SQL Installation Guide.

10.9.5.5.3 Installing Query Server

You can use this method to add Query Server whether Oracle Big Data SQL was installed using Oracle Big Data Appliance tools (Mammoth or bdacli), or the standalone Big Data SQL installer.

In summary, download the Query Server bundle from edelivery.oracle.com and execute the run file it contains. Then use the jaguar utility to add Query to the existing Big Data SQL installation.

Before you start, ensure that the prerequisites described in the previous section are met.

The Query Server installation is on Big Data Appliance only. No change is required to the database side of the Oracle Big Data SQL installation.

There are two files to download. These are the two parts of the Query Server bundle:

• V982741-01_1of2.zip
  V982741-01_2of2.zip

Note:

Do not download V982738-01.zip. This package is not needed in order to add Query Server.

1. Log on to the Oracle Software Delivery Cloud.
2. Search for “Oracle Big Data SQL.”
3. Select Oracle Big Data SQL 4.0.0.0 for Linux x86-64.
   The actual version available may be greater than 4.0.0.0 n.n.n. The same bundle is compatible with all supported Hadoop clusters.
4. Agree to the Oracle Standard Terms and Restrictions. Then you can download the bundle.
5. Copy the two zip files to the node that hosts Configuration Manager. This is usually Node3. Choose any location in the file system.
6. Log in as root and unzip the two files
   You will also see that both files contain the script join.sh. Run this script in order to assemble the bundle You can use the copy of join.sh from either zip file.

   $ unzip -j -o V982741-01_1of2.zip
   Archive: V982741-01_1of2.zip 
     inflating: BDSQLQS82d323d472f5c4666e1a7e48cd2d75b9-00 
     inflating: join.sh 
     inflating: readme.1st 
   
   $ unzip -j -o V982741-01_2of2.zip
   Archive: V982741-01_2of2.zip 
     inflating: BDSQLQS82d323d472f5c4666e1a7e48cd2d75b9-01 
     inflating: join.sh 
     inflating: readme.1st
     
   $ ./join.sh
   Re-assembling Big Data SQL Query Server bundle
   Detected files:
   BDSQLQS82d323d472f5c4666e1a7e48cd2d75b9-00
   BDSQLQS82d323d472f5c4666e1a7e48cd2d75b9-01
   Joining 2 files
   BigDataSQL-4.0.0-QueryServer.zip successfully created !!!

   This join creates a new bundle called BDSExtra-4.0.0-QueryServer.zip. Unzip the newly-created bundle to extract the QueryServer run file and then execute the run file.

   # unzip BDSExtra-4.0.0-QueryServer.zip
   ...
   # ./BDSExtra-4.0.0-QueryServer.run

7. Edit /opt/oracle/BDSjaguar/bds-config.json.
   In the edgedb section of the configuration file, use JSON syntax to identify edge node where you want Query Server to be installed and also list Hive databases that you want the Query Server database to synchronize with.

   "edgedb": {
   "node" : "dbnode.domain.com",
   "enabled" : "true",
   "sync_hive_db_list" : "my_hive_db_1,my_hive_db2"
   }

8. In /opt/oracle/BDSjaguar, invoke the jaguar installer with the reconfigure operator and the configuration filename as parameters.

   # ./jaguar reconfigure bds-config.json
       

   Query Server will be installed on the edge node.

This completes the Query Server installation.

10.9.5.6 Managing the Oracle Big Data SQL Service

When Oracle Big Data SQL is installed on the appliance, its bd_cell service is installed and started on all DataNodes of the Hadoop cluster.

The Oracle Big Data Appliance bdacli utility provides operations to manage the Oracle Big Data SQL service on each DataNode and also cluster-wide.

For the instance of the service on a single DataNode:

bdacli {start | stop | restart | status} big_data_sql_server <node name>

For all instances within the cluster:

bdacli {start | stop | restart | status} big_data_sql_cluster

Note:

The bdacli utility also provides # bdacli {enable | disable} big_data_sql to install or uninstall the service cluster-wide. If you have customized the installation by adding non-default configuration parameters, be aware that these commands will overwrite your custom configuration with the default configuration. If you use them, first be sure that you have archived a copy of your custom configuration file so that you can use the Jaguar reconfigure operation to restore your customizations.

See Also:

The bdacli reference in this guide describes bdacli operations available for managing Oracle Big Data SQL and other services.

10.9.5.7 Uninstalling Oracle Big Data SQL

To completely uninstall Oracle Big Data SQL, remove the software from both Oracle Big Data Appliance and from the Oracle Database system.

Uninstalling from the Oracle Database System

Use the bds-database-install.sh script utility to install/uninstall the database side of Oracle Big Data SQL.

On the Oracle Database server (or on each node of a multinode database) run bds-database-install.sh with the --uninstall parameter.

$ /bds-database-install.sh --uninstall --crs=false  

Note:

If Grid is not running on this database node, or, if the database does not use Grid (CRS/ASM), then include the optional --crs parameter and set it to false.

Also note that the --uninstall-as-primary and --uninstall-as-secondary parameters from previous releases are deprecated in this release. It is no longer necessary to differentiate between primary and secondary clusters in the uninstall process.

Uninstalling from Oracle Big Data Appliance

Use bdacli to disable (uninstall) Oracle Big Data SQL on Oracle Big Data Appliance.

# bdacli disable big_data_sql

This command uninstalls the software from all DataNodes in the entire cluster.

10.9.6 Setting Up and Configuring HBase

HBase is included in the Oracle Big Data Appliance installation, but additional steps are required before you can use it.

Use Configuration Manager to complete the setup of HBase. See Configuring HBase via Cloudera Manager on Oracle Big Data Appliance V2.2 and higher (Doc ID 1588891.1) in My Oracle Support for instructions.

10.10 Installing Other Approved Software

In addition to the optional software included with the Oracle Big Data Appliance deployment bundle, there are other applications that are tested and approved for installation on the appliance.

• Add Support for Oracle Enterprise Manager Cloud Control

• Installing Cloudera Data Science Workbench

If you install software that is not on this list or has not been otherwise approved for use on Oracle Big Data Appliance, then be sure to follow the guidelines described in the Oracle Big Data Appliance Licensing Information User Manual. Also check the licensing manual to determine the party that is responsible for support.

10.10.1 Add Support for Oracle Enterprise Manager Cloud Control

Oracle Big Data Appliance supports these versions of the Enterprise Manager System Monitoring plug-in: 13.2 PG, 13.2, 13.1, and 12.c. You can install the Enterprise Manager System Monitoring plug-in for Oracle Big Data Appliance in an Oracle Enterprise Manager Cloud Control installation on the same network.

Installing the Oracle Enterprise Manager Cloud Control Plug-In

1. Install the preliminary patches (if required for the plug-in version you choose).

   To enable system monitoring over HTTPS for either the 13.1 or 12.c plug-in, you must first install the patches listed in the table below. You can download these patches from the Oracle Automated Release Updates (ARU) site. the 13.2 and 13.2PG plug-ins do not require additional patches.

   Enterprise Manager System Monitoring Plug-In	Patch Prerequisites for Support of the Plug-In on Oracle Big Data Appliance
   13.2 and 13.2 PG Enterprise Manager System Monitoring Plug-Ins. For Oracle Big Data Appliance, the specific 13.2 PG version is Enterprise Manager for Big Data Appliance (13.2.2.0.0).	No patches needed.
   13.1 Enterprise Manager System Monitoring Plug-In	OMS side patch: 23294904 Agent plugin : 23294778 Discovery plugin : 23294785
   12.c Enterprise Manager System Monitoring Plug-In	OMS side patch: 23218275 BDA Plugin Discovery OH patches: 23218115 and 23218109.

2. Install the Enterprise Manager System Monitoring plug-in.

   For instructions on installing the version 13.1 plug-in, see the Enterprise Manager System Monitoring Plug-in Installation Guide for Oracle Big Data Appliance. You can use these same instructions to install version 13.2 and 13.2 PG.

   For instructions on installing the 12.c plug-in, see this My Oracle Support Document:Download/Deployment of the 12c EM May Bundle Patch to Support Discovery/Monitoring of Cloudera Manager Admin Console HTTPS URLS (Doc ID 2148611.1).

What’s New in 13.2 PG for Oracle Big Data Appliance?

The 13.2 PG plug-in for Oracle Big Data Appliance adds support for the following targets:

• Sentry

• Sqoop2

• Key Management Service

• Key Trustee Service

• Cluster Security

See Also:

See the following note in My Oracle Support for a cross-reference of Oracle Big Data Appliance releases and compatible Oracle Enterprise Manager plug-ins.

Oracle Enterprise Manager Cloud Control Big Data Appliance Plug-In Compatibility Matrix for BDA V4.5 and Higher (Doc ID 2267589.1)

10.10.2 Installing Cloudera Data Science Workbench

Cloudera Data Science Workbench (CDSW) can be installed on Oracle Big Data Appliance.

You can install Cloudera Data Science Workbench on Oracle Big Data Appliance edge nodes running Oracle Linux 7. These can be either Mammoth-managed or unmanaged edge nodes.

The installation procedures are available on Cloudera’s documentation website: https://www.cloudera.com/documentation.html

Note:

Cloudera Data Science Workbench is not included in the Oracle Big Data Appliance license and is not covered under Oracle Big Data Appliance support. See the Cloudera website for licensing and support information.

This product is not included in the Oracle Big Data Appliance deployment bundle. You can download the installation package from the Cloudera website.

See Also:

• Cloudera’s description of Cloudera Data Science Workbench at https://www.cloudera.com/products/data-science-and-engineering/data-science-workbench.html

• My Oracle Support (MOS) note 2177796.1, which describes how to convert a cluster node into a Mammoth-managed edge node.

10.11 Reinstalling the Base Image

You may need to reinstall the base image if you want to return Oracle Big Data Appliance to its original state, or you replaced a failed server.

The operating system and various utilities are factory-installed on Oracle Big Data Appliance, as described in "Oracle Big Data Appliance Management Software". Mammoth automatically upgrades the base image as necessary before upgrading the Oracle Big Data Appliance software to a newer version.

You can reimage all or part of a rack. However, all the servers in a cluster must have the same image. Follow the appropriate instructions:

• Reimaging a Single Oracle Big Data Appliance Server

• Reimaging an Oracle Big Data Appliance Rack

• Reimaging an Oracle Big Data Appliance Cluster

10.11.1 Reimaging a Single Oracle Big Data Appliance Server

Follow this procedure to reimage one server.

Caution:

If you reimage a server, then all files and data are erased.

To reinstall the base image on one server:

1. Download the base image patch from My Oracle Support or Oracle Automated Release Updates (ARU), and copy it to the server being reimaged.

   Caution:

   Use the most recent 4.x version of the base image. Do not use the version included in the Mammoth bundle.

   See "Downloading the Mammoth Software Deployment Bundle." You can take the same basic steps to download the base image patch from My Oracle Support.

2. If you are reimaging the server to the current customer settings, verify that /opt/oracle/bda/rack-network.json and /opt/oracle/bda/cluster-network.json reflect the intended network configuration. If it does not, then generate a new set of files using Oracle Big Data Appliance Configuration Generation Utility. See "Generating the Configuration Files."

   (Note that use of /opt/oracle/bda/network.json is still supported, but not recommended.)

3. Ensure that 4 GB or more disk space are free in the partition:

   $ df -h
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/md2              161G   23G  130G  15% /
   /dev/md0              194M   40M  145M  22% /boot
   tmpfs                  24G     0   24G   0% /dev/shm
   /dev/sda4             1.7T  197M  1.7T   1% /u01
   /dev/sdb4             1.7T  197M  1.7T   1% /u02
   /dev/sdc1             1.8T  199M  1.8T   1% /u03
   /dev/sdd1             1.8T  199M  1.8T   1% /u04
        .
        .
        .

4. Unzip the downloaded base image ZIP file. The following example shows a partial listing of the output to the command line:

   # unzip p<number>_<version>_Linux-x86-64.zip
   Archive: p<number>_Linux-x86-64.zip
   creating: BDABaseImage-ol7-<version>_RELEASE/
   creating: BDABaseImage-ol7-<version>_RELEASE/Extras/
   creating: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/
   extracting: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/orabalancer-<version>-h2.zip
   inflating: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/orabalancer-<version>-h2.noarch.rpm
   extracting: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/orabalancerdemo-<version>-h2.zip
   creating: BDABaseImage-ol7-<version>_RELEASE/Extras/CELL/
   inflating: BDABaseImage-ol7-<version>_RELEASE/Extras/CELL/bd_cell-<version>_LINUX.X64_160506.11
   ...
   

5. Change to the subdirectory created in the previous step:

   # cd BDABaseImage-ol7-<version>_RELEASE
    

6. Log on to the server to be reimaged and run the makebdaimage command. The the example below show the command to reimage Node 4 (including the internal USB) from the base image to the custom settings in the configuration files. Comma-separate rack-network.json and cluster-network.json and submit them as a single parameter.

   # ./makebdaimage -usbint BDABaseImage-<version>_RELEASE.iso /opt/oracle/bda/rack-network.json,/opt/oracle/bda/cluster-network.json 4

   Note that submitting network.json as the file parameter still works, but is no longer recommended:

   # ./makebdaimage -usbint BDABaseImage-<version>_RELEASE.iso /opt/oracle/bda/network.json 4
   

7. If the makebdaimage command succeeds without errors, then restart the server.

10.11.2 Reimaging an Oracle Big Data Appliance Rack

Follow this procedure to reimage an entire rack.

Caution:

If you reimage an entire rack, then all clusters, files, and data on the rack are erased. Reimaging is not required for a software upgrade.

To reinstall the base image on all servers in a rack:

1. If the Oracle Big Data Appliance software was installed previously on the rack, then save the /opt/oracle/BDAMammoth/cluster_name-config.json file to a safe place outside Oracle Big Data Appliance.

2. Download the most recent base image patch from My Oracle Support or Oracle Automated Release Updates (ARU), and copy it to the first (bottom) server of the rack being reimaged.

   See "Downloading the Mammoth Software Deployment Bundle." You can take the same basic steps to download the base image patch from My Oracle Support.

   Caution:

   Use the most recent version of the base image. Do not use the version included in the Mammoth bundle.

3. Establish an SSH connection to the first server and log in as root.

4. If you are reimaging to existing customer network settings, then verify that /opt/oracle/bda/network.json reflects the intended network configuration. If it does not, then generate a new file using Oracle Big Data Appliance Configuration Generation Utility.

5. Ensure that passwordless SSH is set up:

   # dcli hostname
   192.nnn.nn.37: bda1node01.example.com
   192.nnn.nn.38: bda1node02.example.com
   192.nnn.nn.39: bda1node03.example.com
        .
        .
        .
   

   This command must run without errors and return the host names of all Oracle Big Data Appliance servers. If not, then follow the steps in "Setting Up Passwordless SSH". Do not continue until the dcli hostname command runs successfully on all servers.

6. Check all Oracle Big Data Appliance servers for hardware issues:

   # dcli bdacheckhw | grep -v SUCCESS
   

7. Resolve any hardware errors and warnings before reimaging the rack.

8. Verify that at least 4 GB are available in the root (/) partition of all servers:

9. # dcli df -h /
   192.nnn.nn.37: Filesystem            Size  Used Avail Use% Mounted on
   192.nnn.nn.37: /dev/md2              161G   21G  132G  14% /
   192.nnn.nn.38: Filesystem            Size  Used Avail Use% Mounted on
   192.nnn.nn.38: /dev/md2              161G   19G  135G  12% /
   192.nnn.nn.39: Filesystem            Size  Used Avail Use% Mounted on
   192.nnn.nn.39: /dev/md2              161G   23G  131G  15% /
        .
        .
        .
   

10. Unzip the downloaded base image ZIP file. For example:

    # unzip p<number>_<version>_Linux-x86-64.zip
    Archive:  p<number>_<version>_Linux-x86-64.zip    
    creating: BDABaseImage-ol7-<version>_RELEASE/   
    inflating: BDABaseImage-ol7-<version>_RELEASE/biosconfig     
    inflating: BDABaseImage-ol7-<version>_RELEASE/makebdaimage    
    extracting: BDABaseImage-ol7-<version>_RELEASE/BDABaseImage-ol6-<version>_RELEASE.md5sum     
    inflating: BDABaseImage-ol7-<version>_RELEASE/reimagerack     
    inflating: BDABaseImage-ol7-<version>_RELEASE/reimagecluster     
    inflating: BDABaseImage-ol7-<version>_RELEASE/BDABaseImage-ol6-<version>_RELEASE.iso      
    .
    .
    .

11. Change to the subdirectory created in the previous step:

    # cd BDABaseImage-ol7-<version>_RELEASE 

12. Complete one of the following procedures:

    • To reimage an Oracle Big Data Appliance that was configured for a customer network to the same customer network settings, execute the ./reimagerack command.

    • To reimage an appliance that still has the factory settings:

      1. Ensure that /opt/oracle/bda/network.json does not exist.

      2. Execute the ./reimagerack command.

    • To restore the factory network settings on a rack configured with custom network settings:

      1. Copy /opt/oracle/bda/network.json to a safe location outside Oracle Big Data Appliance.

      2. Ensure that /opt/oracle/bda/BdaShip.json exists.

      3. Reimage the rack:

         ./reimagerack deploy ship
         

13. Run Mammoth.

See Also:

• "Generating the Configuration Files"

• reimagerack for complete syntax of the reimagerack command.

• "Installing the Software" for instructions on running Mammoth.

10.11.3 Reimaging an Oracle Big Data Appliance Cluster

Follow this procedure to reimage a group of servers that the Mammoth utility has deployed as a single cluster. This can only be used on a group of servers that has been deployed as a single cluster using Mammoth. You can either reapply the network settings that are in currently in effect or revert to factory settings.

Caution:

If you reimage a cluster, then all files and data on the cluster are deleted. Reimaging is not required for a software upgrade.

Mandatory Prerequisites

You must rename the files <rack_name>-rack-network.json and <cluster_name>-cluster-network.json generated by the Oracle Big Data Appliance configuration utility to rack-network.json and cluster-network.json , respectively. In other words, truncate the rack name and cluster name prefixes as well as the first hypen. The only purpose of those prefixes is to enable you to distinguish one cluster or rack from another in the output of the configuration utility.

Important:

Before reimaging, /opt/oracle/bda/rack-network.json and /opt/oracle/bda/rack-network.json must exist on all nodes of the cluster.

To reinstall the base image on all servers in a cluster:

Tip:

The reimagecluster utility creates an installer on the internal USB drive of each server in the rack, initializes the installation, and reboots each server. While the servers are down for reboot, you can log on to Integrated Lights Out Manager (ILOM) on each server to monitor progress:

<server name>-ilom.<domain>.com

Click on Remote Control -> Redirection -> Launch Remote Console to launch a Java console where you can monitor the re-imaging.

1. If the Oracle Big Data Appliance software was installed previously on the rack, then save the/opt/oracle/BDAMammoth/mammoth-<rack_name>.params file (or /opt/oracle/BDAMammoth/<cluster_name>-config.json) to a safe place outside Oracle Big Data Appliance. If the cluster has been upgraded, the file may exist in /opt/oracle/BDAMammoth/previous-BDAMammoth directory.

2. Download the latest Oracle Big Data Appliance Base Image tarball from My Oracle Support to a temporary directory. Copy it onto the first (bottom) server of the cluster to be re-imaged.

3. If re-imaging to your current settings, verify that /opt/oracle/bda/rack-network.json andcluster-network.json reflect the intended network configuration. If they do not, then generate new files using Oracle Big Data Appliance Configuration Generation Utility.

4. Establish an SSH connection to the first server and log in as root.

5. Ensure that password-less SSH has been setup on the cluster (dcli -C hostname should run without errors). If it has not been setup, run setup-root-ssh -C.

   # setup-root-ssh -C 
   Enter root password: 
   ... 
   ... 
   setup-root-ssh succeeded

   dcli -C hostname must run without errors and return the host names of all Oracle Big Data Appliance servers in the cluster. If not, then follow the steps in "Setting Up Passwordless SSH". Do not continue until the dcli -C hostname command runs successfully on all servers.

6. Ensure that there is at least 4 GB free in the / partition of all servers:

   # dcli -C df -h /
   10.xxx.xxx.xx: Filesystem      Size  Used Avail Use% Mounted on 
   10.xxx.xxx.xx: /dev/md2        459G   49G  387G  12% / 
   10.xxx.xxx.xx: Filesystem      Size  Used Avail Use% Mounted on
   ...

7. Unzip the downloaded base image ZIP file. For example:

   # unzip p<number>_<version>_Linux-x86-64.zip
   Archive: p<number>_<version>_Linux-x86-64.zip
   creating: BDABaseImage-ol7-<version>_RELEASE/
   creating: BDABaseImage-ol7-<version>_RELEASE/Extras/
   creating: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/
   extracting: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/orabalancer-<version>-h2.zip
   inflating: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/orabalancer-<version>-h2.noarch.rpm
   extracting: BDABaseImage-ol7-<version>_RELEASE/Extras/BALANCER/orabalancerdemo-<version>-h2.zip
   creating: BDABaseImage-ol7-<version>_RELEASE/Extras/CELL/
   inflating: BDABaseImage-ol7-<version>_RELEASE/Extras/CELL/bd_cell-<version>_LINUX.X64_160506.11   
   ...
   

8. Change to the subdirectory created in the previous step:

   # cd BDABaseImage-ol7-<version>_RELEASE 

9. Use one of the following procedures to re-image the entire Oracle Big Data Appliance. The first alternative is to retain your existing configuration . The second is to restore to factory settings (prior to Oracle Big Data Appliance Configuration Utility setup) .

   Reimage and Retain Existing Network Settings

   1. At /opt/oracle/bda/, check that both network.json and BdaShip.json are listed.

      # ls *.json
      network.json BdaShip.json

      If network.json is not listed, locate the customer-specific network.json file (the one that describes the configuration you want to keep). First check to ensure that this file defines correct network settings. If it does not, generate a new file using Oracle Big Data Appliance Configuration Generation Utility. Then copy the file to /opt/oracle/bda and rename it to network.json.

   2. Use dcli to copy network.json to all nodes of the cluster.

      #  dcli -C -f /opt/oracle/bda/network.json -d /opt/oracle/bda/

   3. Execute the ./reimagecluster command.  Partial output of the command is shown below.

      # ./reimagerack
      Passwordless SSH set up to all hosts : xxx.xxx.xx.xx xxx.xxx.xx.xx xxx.xxx.xx.xx xxx.xxx.xx.xx xxx.xxx.xx.xx 
       xxx.xxx.xx.xx 
      Using /tmp/BDABaseImage-ol7-<version>_RELEASE/BDABaseImage-ol7-<version>_RELEASE.iso to re-image
      BDABaseImage-ol7-<version>_RELEASE.iso: OK
      Re-imaging to existing customer network settings
      
      FINAL CONFIRMATION : All of the following hosts will be completely erased, re-imaged and initialized including resetting of the iloms: xxx.xxx.xx.xx xxx.xxx.xx.xx xxx.xxx.xx.xx xxx.xxx.xx.xx xxx.xxx.xx.xx xxx.xxx.xx.xx
      Proceed [y/n]?
      y
      Running hardware checks on all nodes...
      ...
      Rebooting all servers  
      Broadcast message from [e-mail] (unknown) at 15:58 ...  
      The system is going down for reboot NOW! 

   To Reimage to Factory Network Settings

   1. As described in Step 1, check that /opt/oracle/bda/rack-network.json and cluster-network.json have been copied to a safe location outside Oracle Big Data Appliance and that /opt/oracle/bda/BdaShip.json exists.

   2. Reimage the cluster to factory-shipped settings with the following command:

      # ./reimagecluster deploy ship

10. Whether you re-imaged with existing network settings or restored to factory settings, the final steps are the same. After reboot from reimagecluster completes, run setup-root-ssh to re-establish a connection to all the servers in the rack. This is necessary so that the dcli command can be run to reboot all the servers again.

    # setup-root-ssh
    Enter root password:
    ...
    setup-root-ssh succeeded

11. Reboot all servers once again using the dcli reboot command.

    # dcli reboot

12. Verify that the files /root/BDA_IMAGING_SUCCEEDED and /root/BDA_REBOOT_SUCCEEDED exist on each server in the rack.

    # dcli ls -lrt /root/BDA_* 

    Check that the date and timestamp of /root/BDA_REBOOT_SUCCEEDED correspond to the time of the reboot in the reimaging process. The file may be zero-byte or may contain some INFO messages.

13. Verify the installed base image version, as in this example:

    # imageinfo
    Big Data Appliance Image Info
    IMAGE_VERSION : <version>
    IMAGE_CREATION_DATE : Wed Mar 12 20:19:54 UTC 2019
    IMAGE_LABEL : BDA_<version>_LINUX.X64_RELEASE
    LINUX_VERSION : Oracle Linux Server release <version>
    KERNEL_VERSION : 2.<version>.el6uek.x86_64
    BDA_RPM_VERSION : bda-<version>-1.el6.x86_64
    OFED_VERSION : OFED-IOV-<version>
    JDK_VERSION : jdk1.<version>-fcs.x86_64

You can now run the Mammoth utility to bring the system up to the latest Oracle Big Data Appliance version.

See Also:

• These MOS notes provides additional details on reimaging:

  • Oracle Big Data Appliance Base Image Version 4.10.0 for New Installations or Reprovisioning Existing Installations on Oracle Linux 7 (Doc ID 2416137.1)
  • Oracle Big Data Appliance Base Image Version 4.10.0 for New Installations or Reprovisioning Existing Installations on Oracle Linux 6 (Doc ID 2364648.1)

• About the Mammoth Utility

10.12 Installing a One-Off Patch

One-off patch bundles provide a fix to specific bugs in one or more releases. You use Mammoth to apply the patch to your cluster

To install a one-off patch from a bundle:

1. Download the patch bundle from the Automated Release Update (ARU) system to a directory such as /tmp on the first node of the Oracle Big Data Appliance cluster.

   The file is named BDA-patch-release-patch.zip. The examples in this procedure use the name BDA-patch-<version>-123456.zip.

2. Unzip the file. For example:

   # unzip BDA-patch-<version>-123456.zip
   Archive:  BDA-patch-<version>-123456.zip
      creating: BDA-patch-<version>-123456/
     inflating: BDA-patch-<version>-123456/BDA-patch-<version>-123456.run 
     inflating: BDA-patch-<version>-123456/README.txt
   

3. Change to the patch directory created in Step 2. For example:

   $ cd BDA-patch-<version>-123456
   

4. Extract the contents of the run file. For example:

   $ ./BDA-patch-<version>-123456.run
   Big Data Appliance one-off patch 123456 for v<version> Self-extraction
    
   Removing existing temporary files
    
   Generating /tmp/BDA-patch-<version>-123456.tar
   Verifying MD5 sum of /tmp/BDA-patch-<version>-123456.tar
   /tmp/BDA-patch-<version>-123456.tar MD5 checksum matches
    
   Extracting /tmp/BDA-patch-<version>-123456.tar to /opt/oracle/BDAMammoth/patches/123456
   Removing temporary files
    
   Please "cd /opt/oracle/BDAMammoth" before running "./mammoth -p 123456"
   

5. Change to the BDAMammoth directory:

   $ cd /opt/oracle/BDAMammoth
   

6. Install the patch. For example:

   $ ./mammoth -p 123456
   

   Mammoth prompts you to choose whether or not you want to load the patch as rolling upgrade. (See "About Rolling Upgrades")

   Alternatively, you can use the bdacli command. See "bdacli".

10.13 Mammoth Software Installation and Configuration Utility

Mammoth is the utility for installing, Oracle Big Data Appliance release software and also for performing upgrades and cluster extensions.

You must log in as root on the first server and change to the /opt/oracle/BDAMammoth directory to use Mammoth. It has this syntax:

./mammoth option [cluster_name] ]

In this command, cluster_name is the name of the cluster You must enter cluster_name in the first command exactly as it appears in (cluster_name-config.json). Afterward, cluster_name defaults to the rack specified in a previous mammoth command.

In a multirack system, you must finish installation on one rack before starting the installation on another rack.

The next section of this guide lists and describes the Mammoth command line options.

10.13.1 Mammoth Options

Use the Mammoth utility to install the Oracle Big Data Appliance software and also to perform upgrades, extend clusters, and run checks on CDH, Kafka, and NoSQL clusters.

The Mammoth command line options are as follows. Note that the -j option available in previous releases has been removed.

-c

Run the Oracle Big Data Appliance cluster checks.

-e newnode1 newnode2 newnode3...

Generates a parameter file for the list of servers to be added to a cluster. It then adds the new node or nodes to the cluster and installs required software on these nodes.

The servers in the list can be in the same rack or multiple racks.

Reboots normally occur only for the new nodes and only at the initial update of the factory image.

The parameter file generated is opt/oracle/BDAMammoth/<cluster_name>-config.json, where <cluster-name> is the name of the cluster being extended. No passwords are included in the parameter file, so you must enter them when running Mammoth. Cluster extension expects the default root password on nodes in the extension, so you should retain that password until the extension has completed. Be sure to change passwords afterward.

On Oracle NoSQL Database clusters, Mammoth prompts for the kind of zone for the new nodes. You can choose from an existing zone, a new primary zone, or a new secondary zone. When adding to an existing zone, Mammoth lists the zones that you can use. When creating a new zone, Mammoth prompts for the zone name and replication factor.

You can use the -r (range) or -s (step) options in conjunction with mammoth -e, as in:

# mammoth -r 1-4 -e newnode1 newnode2 newnode3...

# mammoth -s 2 -e newnode1 newnode2 newnode3...

-h

Displays command Help including command usage and a list of steps.

-i cluster_name

Runs all mandatory steps on the cluster, equivalent to -r 1-18 for a full rack. Use this option when configuring a new rack or adding a group of servers to a cluster.

-l

Lists the steps of the Mammoth installation.

-p

Upgrades the software on the cluster to the current version or installs a one-off patch.

You can use the -r (range) or -s (step) options in conjunction with mammoth -p, as in:

# mammoth -r 1-4 -p

# mammoth -s 2 -p

-r n-N

Runs steps n through N of Mammoth while no errors occur.

-s n

Runs step n. Enter cluster_name to identify another cluster on the same rack. The default is the current cluster. See the -e option.

Note:

With both -s and -r, the proper sequence of install, upgrade, or extension steps is enforced. You cannot use these options to bypass steps.

-v

Displays the version number of the Mammoth.

10.13.2 Mammoth Installation Steps

Mammoth performs the following sequence of steps in an Oracle Big Data Appliance release installation.

Step 1   PreinstallChecks

This step performs several tasks:

• Validates the configuration files and prompts for the passwords.

• Sets up a Secure Shell (SSH) for the root user so you can connect to all addresses on the administrative network without entering a password.

• Sets up passwordless SSH for the root user on the InfiniBand network.

• Generates /etc/hosts from the configuration file and copies it to all servers so they use the InfiniBand connections to communicate internally. The file maps private IP addresses to public host names.

• Checks the network timing on all nodes. If the timing checks fail, then there are unresolved names and IP addresses that will prevent the installation from running correctly. Fix these issues before continuing with the installation.

This step also performs a variety of hardware and software checks. A failure in any of these checks causes the Mammoth to fail:

• The ARP cache querying time is 2 seconds or less.

• All server clocks are synchronized within 10 seconds of the current server.

• All servers succeeded on the last restart and generated a /root/BDA_REBOOT_SUCCEEDED file.

• The bdacheckhw utility succeeds.

• The bdachecksw utility succeeds.

Step 2   SetupAnsible

This step configures Ansible on all nodes. After this step is completed, Ansible can deploy the software.

Ansible is an open-source software provisioning, configuration management, and application-deployment tool. BDA Mammoth uses ansible to deploy the cluster software and configuration.

Step 3   UpdateBaseImage

Installs the most recent Oracle Big Data Appliance image and system parameter settings. The system automatically restarts if the kernel is upgraded.

Step 4   PrepareBaseImage

• Copies third-party licenses to /opt/oss/src/OSSLicenses.pdf on every server, as required by the licensing agreements.

• Copies third-party software source code to /opt/oss/src/ on every server, as required by the licensing agreements.

  Mammoth does not copy the source code to Oracle NoSQL Database clusters.

• Creates the hdfs and mapred users, and the hadoop group. It also creates the oracle user and the dba and oinstall groups.

  The various packages installed in later steps also create users and groups during their installation.

  See Also:

  Oracle Big Data Appliance Software User's Guide for more information about users and groups.

• The NameNode data is copied to multiple places to prevent a loss of this critical information should a failure occur in either the disk or the entire node where they are set up.

Step 5   SetupMySQL

Installs and configures MySQL Database. This step creates the primary database and several databases on Node3 for use by Cloudera Manager. It also sets up replication of the primary database to a backup database on Node2.

Mammoth does not install MySQL Database on Oracle NoSQL Database clusters.

Step 6   SetupKDC

Configures the Kerberos Key Distribution Center (KDC) that is used to set up security on the cluster.

Step 7   InstallHadoop

Installs all packets in Cloudera's Distribution including Apache Hadoop (CDH) and Cloudera Manager. It then starts the Cloudera Manager server on Node3 and configures the cluster.

Mammoth does not install CDH or Cloudera Manager on Oracle NoSQL Database clusters.

Step 8   StartHadoopServices

Starts the agents on all nodes and starts all CDH services. Also enables HTTPS for Cloudera Manager, Hue, and Oozie if HTTPS/ Network Encryption was enabled.

After this step, you have a fully functional Hadoop installation.

Cloudera Manager runs on port 7180 of Node3. You can open it in a browser, for example:

http://bda1node03.example.com:7180

In this example, bda1node03 is the name of Node3 and example.com is the domain. The default user name and password is admin. You are prompted to change the password at the end of this step.

Note that Mammoth does not install or start CDH services on Oracle NoSQL Database clusters.

Step 9   InstallBDASoftware

Installs the server-side components of Oracle Big Data Connectors, if this option was selected in Oracle Big Data Appliance Configuration Generation Utility. Oracle Big Data Connectors must be licensed separately. Optional.

Installs the server-side components of Oracle Big Data SQL, if this option was selected in Oracle Big Data Appliance Configuration Generation Utility. Oracle Big Data SQL must be licensed separately.

Installs Oracle NoSQL Database on clusters allocated to its use. Enterprise Edition requires a separate license.

Step 10   SetupHadoopSecurity

Actions performed in this step are based upon the choices you made in the Oracle Big Data Appliance Configuration Utility prior to installation.

• Configures HTTPS/ Network Encryption if this option was selected.

• Configures HDFS Transparent Encryption if this option was selected.

  Mammoth sets up the The Navigator Key Trustee services required by HDFS Transparent Encryption based on whether you chose local or external Key Trustee servers in the configuration utility prior to installation:

  • If you had selected Setup Key Trustee Servers on the BDA in the configuration utility, Mammoth will setup up active and passive Key Trustee servers locally on the Oracle Big Data Appliance.

  • If you chose to use external Key Trustee Servers, the connection to the external servers is configured in this setup. (Note that you must have already configured the external Navigator Key Trustee Servers as described in MOS Doc ID 2112644.1)

• Configures Kerberos authentication if this option was selected. There are no prerequisites if you set up the key distribution center on Oracle Big Data Appliance. Otherwise, see "Installation Prerequisites."

Step 11   SetupASR

Installs and configures Auto Service Request (ASR). Optional.

This step does the following:

• Installs the required software packages

• Configures the trap destinations

• Starts the monitoring daemon

To activate the assets from ASR Manager, see "Verifying ASR Assets".

Note:

For this step to run successfully, the ASR host system must be up with ASR Manager running and configured properly. See Setting Up Auto Service Request.

Step 12   CleanupInstall

Performs the following:

• Changes the Cloudera Manager password if specified in the Installation Template.

• Deletes temporary files created during the installation.

• Copies log files from all nodes to subdirectories in /opt/oracle/bda/install/log.

• Runs cluster verification checks, including TeraSort, to ensure that everything is working properly. It also generates an install summary. All logs are stored in a subdirectory under /opt/oracle/bda/install/log on Node1.

Step 13   CleanupSSHroot (Optional)

Removes passwordless SSH for root that was set up in "PreinstallChecks".

10.14 Oracle Big Data Appliance Base Imaging Utilities

The following imaging utilities are distributed in the base image bundle.

To run these utilities, you must be logged in as root.

• makebdaimage

• reimagecluster

• reimagerack

10.14.1 makebdaimage

Reinstalls the base image on a single server, a cluster, or an entire rack. Both reimagecluster and reimagerack call this utility.

The makedbaimage utility has this syntax:

makebdaimage [--usb | --usbint] [--noiloms] /path/BDABaseImage-version_RELEASE.iso /path/BDdaDeploy.json target_servers

Options

--usb | --usbint

Identifies the USB port that will be used for reimaging. Use --usb for the external USB drive, or --usbint for the internal drive. The internal USB drive contains a full Linux installation.

To use the --usbint option, you must be logged in to the target server; otherwise, you reimage the server using the wrong network information.

--noiloms

The reimaging process does not alter the Oracle ILOMs on the target servers.

target_servers

One or more servers where the image will be installed, which you specify using one of these formats:

• node_number

  Identifies one server for reimaging. Enter the position of the server in the rack, from 1 to 18, counting from bottom to top.

• from.json to.json

  Identifies the current configuration (from.json) and the desired configuration (to.json). The JSON files can be either the factory configuration in BdaShip.json or a custom configuration in network.json. You can find them in /opt/oracle/bda on configured servers.

• to.json: Concatenation of JSON files similar to network.json and BdaShip.json, but containing an extra ETH0_MACS array entry. The first entry in the concatenated file with a matching eth0_mac entry is used when reimaging. The to.json file is used as is.

10.14.2 reimagecluster

Reimages all servers in the cluster in parallel using dcli and makebdaimage.

The reimagecluster utility has this syntax:

reimagecluster [--no-iloms] 

Prerequisites

• Verify that the following command returns the list of servers in the cluster:

  $ dcli -C hostname
  

• Ensure that exactly one BDABaseImage-version_RELEASE*.iso file is in the current directory.

Options

--no-iloms

The reimaging process does not alter the Oracle ILOMs on the target servers.

10.14.3 reimagerack

Reimages all servers in the rack in parallel using dcli and makedbaimage.

The reimagerack utility has this syntax:

reimagerack [--no-iloms] [--no-macs] [--hosts n1, n2...] [from.json [to.json]]

Prerequisites

• Verify that the following command returns the list of servers in the rack:

  $ dcli -hostname
  

• Ensure that exactly one BDABaseImage-version_RELEASE*.iso file is in the current directory.

Options

--no-iloms

The reimaging process does not alter the Oracle ILOMs on the target servers.

--no-macs

The utility does not retrieve the server MAC addresses. Instead, it uses the InfiniBand cabling. Use this option only when restoring the factory settings; you must include both JSON files (from.json and to.json) in the command line.

--hosts n1, n2...

Restricts reimaging to the servers identified by a comma-delimited list of host names or IP addresses, whichever one dcli accepts. All servers must be in the list of target servers identified by the dcli -t command.

This option enforces the --no-macs option, so its restrictions also apply.

from.json

The full path to the current configuration file, either BdaShip.json or network.json.

This option defaults to /opt/oracle/bda/network.json. If network.json is missing, then the option defaults to /opt/oracle/bda/BdaShip.json. The servers must already be set to the values in the JSON file used by reimagerack.

to.json

The full path to the new network configuration in effect after reimaging. It can be either a network.json or a BdaShip.json file.