2 Installing the Hadoop Side of Oracle Big Data SQL

Services Running

These Apache Hadoop services must be running on the cluster.

• HDFS

• YARN

• Hive

You do not need to take any extra steps to ensure that the correct HDFS and Hive clients URLs are specified in the database-side installation bundle.

Packages

The following packages must be pre-installed on all Hadoop cluster nodes before installing Oracle Big Data SQL.

• Oracle JDK version 1.7 or later

• dmidecode

• net-snmp, net-snmp-utils

• perl

  PERL LibXML – 1.7.0 or higher, e.g. perl-XML-LibXML-1.70-5.el6.x86_64.rpm

  perl-libwww-perl, perl-libxml-perl, perl-Time-HiRes, perl-libs, perl-XML-SAX, perl-Env

The Java JDK is available for download on the Oracle Technology Network.

The yum utility is the recommended method for installing these packages:

yum -y install dmidecode
yum -y install net-snmp net-snmp-utils
yum -y install perl perl-libs
yum -y install perl-Time-HiRes perl-libwww-perl
yum -y install perl-libxml-perl perl-XML-LibXML perl-XML-SAX
yum -y install perl-Env
yum -y localinstall <JDK RPM that you downloaded>

System Tools

• curl

• gcc

• libaio

• rpm

• scp

• tar

• unzip

• wget

• yum

• zip

yum install -y libaio gcc 

Environment Settings

The following environment settings are required prior to the installation.

• NTP enabled

• Ensure that /usr/java/default exists and is linked to the appropriate Java version (if $JAVA_HOME does not exist).

• The path to the Java binaries must exist in /usr/java/latest.

• The installation process requires Internet access in order to download some packages from Cloudera or Hortonworks sites. If a proxy is needed for this access, ensure that the following Linux environment variables are properly set:

  • http_proxy and https_proxy

  • no_proxy

    Set no_proxy to include the following: "localhost,127.0.0.1,<Comma—separated list of the hostnames in the cluster (in FQDN format).>".

• On Cloudera CDH, clear any proxy settings in Cloudera Manager administration before running the installation. You can restore them after running the script that creates the database-side installation bundle (bds-database-create-bundle.sh).

Python 2.7 (for the Oracle Big Data SQL Installer)

The Oracle Big Data SQL installer requires Python 2.7 locally on the node where you run the installer. This should be the same node where the cluster management service (CM or Ambari) is running.

If any version of Python 2.7.x is already installed, you can use it to run the Oracle Big Data installer.

On Oracle Linux 5, add Python 2.7 as a secondary installation. On Oracle Linux 6, both Python 2.6 and 2.7 are pre-installed and you should use the provided version of Python 2.7 for the installer. Check whether the default interpreter is Python 2.6 or 2.7. To run the Oracle Big Data SQL installer, you may need to invoke Python 2.7 explicitly. On Oracle Big Data Appliance, SCL is installed so you can use to enable version 2.7 for the shell as in this example:

# scl enable python27 "./setup-bds install bds-config.json"

Below is a procedure for adding the Python 2.7.5 as a secondary installation.

# yum install -y openssl-devel

# pyversion=2.7.5
# cd /tmp/
# mkdir py_install
# cd py_install
# wget https://www.python.org/static/files/pubkeys.txt
# gpg --import pubkeys.txt
# wget https://www.python.org/ftp/python/$pyversion/Python-$pyversion.tgz.asc
# wget https://www.python.org/ftp/python/$pyversion/Python-$pyversion.tgz
# gpg --verify Python-$pyversion.tgz.asc Python-$pyversion.tgz
# tar xfzv Python-$pyversion.tgz
# cd Python-$pyversion
# ./configure --prefix=/usr/local/python/2.7
# make
# mkdir -p /usr/local/python/2.7
# make install
# export PATH=/usr/local/python/2.7/bin:$PATH

If Oracle Big Data SQL is Already Installed

If Oracle Big Data SQL is already installed, please read Upgrading From a Prior Release of Oracle Big Data SQL before proceeding.

On the Hadoop side, you manually install the software on the cluster management server only. The installer uses CDH or Ambari to analyze the cluster configuration and automatically deploys and installs Oracle Big Data SQL on all nodes where it is required.

• Extract the files from BigDataSQL-<Hadoop_distribution>-<version>.zip archive. Then, configure bds-config.json and run setup-bds with the install argument. This installs Oracle Big Data SQL on the cluster management server as well as on all Hadoop DataNodes in the cluster.

• Run the database bundle creation script, bds-database-create-bundle . This generates the database bundle file that you will run on the Oracle Database server in order to install Oracle Big Data SQL on the Oracle Database server.

• Check the parameters in the database bundle file and adjust as needed.

After you have checked and (if necessary) edited the database bundle file, copy it over to the Oracle Database server and run it as described in Installing on the Oracle Database Server

Install Oracle Big Data SQL on the Cluster Management Server

To install Big Data SQL on the cluster management server:

1. Copy the appropriate zip file (BigDataSQL-<Hadoop_distribution>-<version>.zip) to a temporary location on the cluster management server (the node where Cloudera Manager or Ambari is running).

2. Unzip file.

   1. For CDH systems, run Patch 25796576 at this point and then proceed with the Oracle Big Data SQL installation.

3. Change directories to BDSSetup .

4. Edit the configuration file.

   Table 2–4 below describes the use of each configuration parameter.

   • For CDH, edit bds-config.json , as shown in this example. Any unused port will work as the web server port.

     {
     "cluster": {
        "name": "cluster1",
        "display_name": "Cluster 1"
     },
     "database":{
        "ip": "10.11.12.13/14"
     },
     "memory": {
        "min_hard_limit": 8192
     },
     "webserver": {
        "port": 80
     }
     }
     

   • For HDP, edit bds-config.json as in this example. Notice that the HDP configuration file does not include the display_name or min_hard_limit parameters.

     {
     "cluster": {
        "name": "cluster1",
     },
     "database":{
        "ip": "10.11.12.13/14"
     },
     "webserver": {
        "port": 80
     }
     }
     

   DATABASE_IP must be the correct network interface address for the database node where you will perform the installation. You can confirm this by running /sbin/ip -o -f inet addr show on the database node.

   Note:

   The next step requires the cluster administrator user ID and password.

5. In the BDSSetup directory, become root and run setup-bds. Pass it the install parameter and the configuration file name (bds-config.json ) as arguments. Note that the cluster management service is restarted in this process.

   [root@myclusteradminserver:BDSSetup] # ./setup-bds install bds-config.json
   

   The script prompts for the cluster management service administrator credentials and then installs Oracle Big Data SQL on the management server and the cluster nodes. The script output terminates with the following message if the installation completed without error.

   BigDataSQL: INSTALL workflow completed.
   

   See Also:

   This is a condensed example. An example of the complete standard output from a successful installation is provided in Oracle Big Data SQL Install/Uninstall/Reconfigure Examples.

Parameters in bds-config.json

The table below explains edits you need to make to bds-config.json for CDH and HDP platforms.

Operations Performed by setup-bds

The table below lists the full set of operations performed by setup-bds .

The syntax for all options is:

# ./setup-bds <option> bds-config.json

For example:

# ./setup-bds install bds-config.json

External Resources Required for the Database-Side Bundle

Running bds-database-create-bundle

Manually Adding Resources if Download Sites are not Accessible to the BDS Database Bundle Creation Script

If one or more of the default download sites is inaccessible from the cluster management server, there are two ways around this problem:

• Download the files from another server first and then provide bds-database-create-bundle.sh with the alternate path as an argument. For example:

  $ ./bds-database-create-bundle.sh --yarn-conf-ws='http://nodexample:1234/config/yarn'
  

• Because the script will first search locally in /bds-database-install for resources, you can download the files to another server, move the files into /bds-database-install on the cluster management server and then run the bundle creation script with no additional argument. For example:

  $ cp hadoop-xxxx.tar.gz bds-database-install/
  $ cp hive-xxxx.tar.gz bds-database-install/
  $ cp yarn-conf.zip bds-database-install/
  $ cp hive-conf.zip bds-database-install/
  $ cd db
  $ ./bds-database-create-bundle.sh
  

Copying the Database Bundle to the Oracle Database Server

Use scp to copy the database bundle you created to the Oracle Database server. In the example below, dbnode is the database server. The Linux account and target directory here are arbitrary. Use any account authorized to scp to the specified path.

For a first-time installation of the current release of Oracle Big Data SQL, copy bds-database-install.zip (the full installation bundle) to each database node.

$ scp bds-database-install.zip oracle@dbnode:/home/oracle

If you are updating the configuration in the existing Oracle Big Data SQL installation on the database servers, copy the smaller configuration update bundle to the database nodes (bds-database-install-config.zip) instead of (bds-database-install.zip).

$ scp bds-database-install-config.zip oracle@dbnode:/home/oracle