5 Adapting to Hadoop Cluster and Oracle Database Changes

Changes that occur to the Hadoop cluster over time, such as network reconfiguration or DataNode services added, moved, or removed, as well as changes on the Oracle Database server, can require corresponding changes to the Oracle Big Data SQL installation.

5.1 Installing on New Hadoop DataNodes After a Cluster Extension

If nodes running the DataNode service are added to the Hadoop cluster, then Oracle Big Data SQL must be installed on these nodes.

Run setup-bds with the extend argument. This operation installs the Oracle Big Data SQL components on nodes where they are required and updates the cells inventory.

# ./setup-bds extend bds-config.json

5.2 Reconfiguring an Existing Oracle Big Data SQL Installation on Database Nodes

When Oracle Big Data SQL is initially installed on the cluster management server, the database-side installation bundles you create contain cluster configuration information that enables Oracle Big Data SQL on the database server to connect to the Hadoop cluster. After a configuration change in the Hadoop cluster, the set of changes must be propagated to the database nodes in order to maintain connectivity with the cluster. To do this, run setup-bds reconfigure bds-config.json on the the cluster management server to capture the changes. Then, bds-database-bundle-creation.sh --reconfigure to create an updated database bundle. And finally, copy the bundle over to each of the database nodes, unpack it, and run bds-database-install.sh --reconfigure.

Perform this procedure if Oracle Big Data SQL is already installed on a Oracle Database node. (To perform a full installation of the software on a database node, do not use this procedure. In that case, see Installing on the Oracle Database Server instead.)

Here are some Hadoop-side changes which require you to reconfigure Oracle Big Data SQL on the database nodes:

  • Changes to the DataNode inventory – DataNodes have be added, moved, or removed.

  • Network security changes, such as a switch from HTTP to HTTPS or vice versa.

  • Port reassignments in the cluster management service (CM or Ambari).

  • Hive changes, such as migration of the service to different node or nodes or a change to the authentication method.

  • Changes to other related Hadoop services, such as HDFS, MapReduce2, and YARN, may also affect connectivity.

Important:

You should perform this task as soon as possible after a cluster-side configuration change, because new or reconfigured DataNodes may be invisible to Oracle Big Data SQL on the database server.

If the DataNodes inventory changes, skip Step 1 below. Instead, make any needed adjustments to bsd-config and run setup-bds extend bds-config.json or setup-bds remove bds-config.json as appropriate. This is a necessary prerequisite and both of these commands implicitly include setup-bds reconfigure bds-config.json.

  1. On the cluster management server, cd to the /BDSSetup directory and run setup-bds as root, using the reconfigure option.

    [root@myclustermgmtserver: BDSSetup] # ./setup-bds reconfigure bds-config.json

    This updates the cluster configuration files that will be incorporated into the new database bundle.

  2. Cd to /BDSSetup/db and recreate the bundle files by executing the bds-database-create-bundle.sh script. Use the --reconfigure switch.

    [root@myclustermgmtserver: db] # ./bds-database-bundle-creation.sh --reconfigure

    This command deletes the old database bundle files and recreates them, using the up-to-date configuration files generated by the previous command.

  3. Copy the new bds-database-install-config.zip bundle file to the oracle home on the database node. Connect using the oracle account on the database node.

    [root@myclustermgmtserver: db] scp bds-database-install-config.zip oracle@dbnode:/home/oracle

  4. Log on to the database node as the oracle user.

  5. Locate the bds-database-install-config.zip file that you copied over from the cluster management server (as described in Creating the Database-Side Installation Bundle).

  6. Unzip the bundle into the directory where Oracle Big Data SQL is already installed. Use the -o switch in order to overwrite the existing installation:

    [oracle@mydbserver: bds-database-install] $ unzip -o bds-database-install-config.zip -d /home/oracle/bds-database-install

  7. Run bds-database-install.sh with the --reconfigure switch. This updates the local configuration on the database node.

    [oracle@mydbserver: bds-database-install] $ ./bds-database-install.sh --reconfigure

5.3 Extending an Existing Installation to New Database Nodes

If you want to install Oracle Big Data SQL on additional Oracle Database nodes at any time after the initial installation of Oracle Big Data SQL, install the full database-side installation bundle (bds-database-install.zip) on these nodes.

Note:

The full bundle contains the required Hadoop client tarballs as well as the Hadoop cluster configuration files. Do not use the bds-database-install-config.zip bundle for a full installation on a database node. bds-database-install-config.zip is for reconfiguring existing Oracle Big Data SQL installations on the database side and does not contain all of files required for a complete installation.

  1. Even if you still have a copy of bds-database-install.zip from a previous installation, you should first regenerate the database bundle as described in Creating the Database-Side Installation Bundle and use the new version. This ensures that the bundle contains the current configuration data.

  2. Deploy and install the bundle on the new database nodes as described in Installing on the Oracle Database Server.