C Oracle Big Data SQL Installation Examples
The following are samples of the console output for the Oracle Big Data SQL installation.
-
Example 1 shows the output of an installation on Oracle Big Data Appliance.
-
Example 2 shows the parallel installation on Hortonworks HDP.
The database side of the installation is the same for Exadata Database Machine and all commodity Oracle Database platforms.
The database side of these scripts shows the installation on a system running Oracle Database 12.1. The output is the same for an installation on Oracle Database 18c and 12.2 systems, except for minor path changes.
Example 1: Installation on Oracle Big Data Appliance
This example shows the Hadoop-side installation where the Hadoop system is Oracle Big Data Appliance. The installation on HDP is similar.
Here are things to note about the example:
- The Big Data Appliance in this case is running the Oracle Big Data Appliance release 4.10. Output may differ slightly on other releases.
- We do not use
sclon this system because on Oracle Big Data Appliance systems running Oracle Linux 6 or 7, Jaguar will automatically usesclif needed. - The
--requestdbparameter takes the name of the database where you will install the other side of Oracle Big Data SQL will be installed after completing the Hadoop-side installation. The database name in this case is orcl. When this parameter is included in the command, Jaguar will a generate a "request key" along with the database-side installation bundle. This key is part of the process that authorizes access to the database. See Important Terms and Concepts for more information about--requestdband about the database authorization process. - There is no JSON configuration file specified on the command line because we are using the default
bds-config.jsonfile. If we used a different configuration file, then the file name would need to be explicitly included in the command. - The output file names are branded with the cluster name, the fully-qualified domain name of the node where the files are generated, and a time stamp. The cluster name in this example is "afghsec."
- The line
BigDataSQL: Object Store support: Enabledindicates that thebds-config.jsonconfiguration file was set up to enable Oracle Big Data SQL access to either Amazon S3 or Oracle OCI. Likewise,DB version support: 18indicates thatbds-config.jsonwas set to enable compatibility with Oracle Database 18c and 12c. The lineWARNING: If there are services in the stale state, they will be restarted !!!represents a default choice in the configuration file. You can configure the installation to bypass restarts of stale services. Other configuration file settings are reported in the output, such as the network connection type:BigDataSQL: Connection type: Infiniband. See the Jaguar Configuration Parameter and Command Reference in this guide for a complete list of configuration options. - Placeholders in brackets are used to fill in hostnames and IP addresses in this example
- You can run these commands as the
oracleuser. However, you will be prompted for therootpassword in order to do the secure copy of files over to the database server.
[root@scaj41bda18 BDSJaguar]# ./jaguar --requestdb orcl install
BigDataSQL: =======================================================================.
BigDataSQL: Oracle Big Data SQL 4.0.0.
BigDataSQL: Starting execution on: Mon May 20 16:52:37 2019.
BigDataSQL: in directory: /opt/oracle/BDSJaguar.
BigDataSQL: and hostname: <node.domain.com>.
BigDataSQL: IP address: <IP address>.
BigDataSQL: DB version support: 18.
BigDataSQL: Object Store support: Enabled.
BigDataSQL: API url: https://<node.domain.com>:7183.
BigDataSQL: Waiting for management server to respond...
BigDataSQL: Configuration json file name: bds-config.json.
BigDataSQL: WARNING: If there are services in the stale state, they will be restarted !!!.
Cloudera Manager user: admin
Password:
BigDataSQL: Authentication succeeded.
BigDataSQL: API version: 17.
BigDataSQL: Cluster name : afghsec.
BigDataSQL: Cluster type : BDA.
BigDataSQL: Cluster version: CDH5.
BigDataSQL: full version: 5.12.1.
BigDataSQL: BDA version : 4.10.0-9.
BigDataSQL: Cluster health check..
BigDataSQL: Service: BIGDATASQL.
BigDataSQL: Verifying service configuration..
BigDataSQL: Connection type: Infiniband.
BigDataSQL: Initialization completed..
BigDataSQL: Copying CSD to Cloudera Manager..
BigDataSQL: Restarting management server...
BigDataSQL: Waiting for management server to respond...
BigDataSQL: Restarting Cloudera Management service..
BigDataSQL: Starting deployment server..
BigDataSQL: Analyzing cluster ....
BigDataSQL: Checking for Big Data SQL requirements..
BigDataSQL: Preparing installation..
BigDataSQL: Installing Big Data SQL agents..
BigDataSQL: Installing Big Data SQL servers..
BigDataSQL: Starting bigdatasql service..
BigDataSQL: reqkey file created: /opt/oracle/BDSJaguar/dbkeys/orcl.reqkey.
BigDataSQL: Restarting stale services..
BigDataSQL: Restarting Cloudera Management service..
BigDataSQL: Big Data SQL task completed..
BigDataSQL: Creating database side bundle..
BigDataSQL: Creating bundle file.
BigDataSQL: Bundle file created: /opt/oracle/BDSJaguar/db-bundles/bds-4.0.0-db-afghsec-190520.1652.zip.
BigDataSQL: INSTALL workflow completed.
Copy the database-side installation bundle to the database node.
[oracle@scaj22bda08 ~]$ scp root@<database host>:/opt/oracle/BDSJaguar/db-bundles/bds-4.0.0-db-afghsec-190520.1652.zip .
root@<host name>'s password:
bds-4.0.0-db-afghsec-190520.1652.zip 100% 870MB 232.9MB/s 00:03Unzip the bundle, execute the run file, and copy the request key file into the install directory created by the run file.
$ unzip bds-4.0.0-db-afghsec-190520.1652.zip
Archive: bds-4.0.0-db-afghsec-190520.1652.zip
inflating: bds-4.0.0-db-afghsec-190520.1652.run
$ ./bds-4.0.0-db-afghsec-190520.1652.run
bds-4.0.0-db-afghsec-190520.1652.run: platform is : Linux
bds-4.0.0-db-afghsec-190520.1652.run: script started at : Mon May 20 17:07:32 PDT 2019
bds-4.0.0-db-afghsec-190520.1652.run: running location : /home/oracle
bds-4.0.0-db-afghsec-190520.1652.run: ORACLE_SID : orcl
bds-4.0.0-db-afghsec-190520.1652.run: ORACLE_HOME : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database
bds-4.0.0-db-afghsec-190520.1652.run: install directory : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/BDSJaguar-4.0.0/afghsec-3-<host FQDN>
bds-4.0.0-db-afghsec-190520.1652.run: extracting...
bds-4.0.0-db-afghsec-190520.1652.run: installer extracted successfully
$ cd $ORACLE_HOME/BDSJaguar-4.0.0/afghsec-3-<FQDN of this Hadoop node>
$ scp root@scaj41bda18:/opt/oracle/BDSJaguar/dbkeys/orcl.reqkey .
root@<host name>'s password:
orcl.reqkey 100% 251 425.6KB/s 00:00
Change to the install directory and run the install operation of bds-database-install.sh.
Notice that in this example, that the default patch check reports that not all prerequisite database patches are installed. This prevented completion of the installation.
$ ./bds-database-install.sh --install --restart-db=yes
bds-database-install: platform is : Linux
bds-database-install: script started at : Mon May 20 17:08:24 PDT 2019
bds-database-install: running location : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/BDSJaguar-4.0.0/afghsec-3-<FQDN of Hadoop node>
bds-database-install: ORACLE_SID : orcl
bds-database-install: ORACLE_HOME : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database
bds-database-install: Validating Database Patches in /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database
bds-database-install: ERROR: Proactive Patches and/or one-off patches are missing.
For more information, please check MOS note:
Oracle Big Data SQL Master Compatibility Matrix (Doc ID 2119369.1)
bds-database-install: log files in /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install
bds-database-install: bds-database-install-16055-* files for more detailed info
bds-database-install: full log in bds-database-install-16055-main-script.log
You can temporarily bypass the database patch check
If you include the --skip-db-patches-check parameter, the installer bypasses the patch check. You will need to install the patches later in order for Big Data SQL to become operational.
Part of the database-side installation must run as root. Notice that the installer generates a script and prompts you to run it as root.
$ ./bds-database-install.sh --install --restart-db=yes --skip-db-patches-check
bds-database-install: platform is : Linux
bds-database-install: script started at : Mon May 20 17:08:52 PDT 2019
bds-database-install: running location : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/BDSJaguar-4.0.0/afghsec-3-<Hadoop node FQDN>
bds-database-install: ORACLE_SID : orcl
bds-database-install: ORACLE_HOME : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database
bds-database-install: Install will affect all pdbs.
bds-database-install: configuration spfile : spfileorcl.ora
bds-database-install: Grid home : not detected
bds-database-install: db resource : orcl
bds-database-install: mta extproc sid : bds_orcl_afghsec
bds-database-install: use of crsctl : false
bds-database-install: Kerberos variables file : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/bigdatasql/clusters/afghsec/bds_kerberos_variables.env
bds-database-install: root shell script : /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-root-script-afghsec.sh
please run as root:
/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-root-script-afghsec.sh
waiting for root script to complete, press <enter> to continue checking.. q<enter> to quitWhen the script finishes, you should see a success message, such as, "bds-database-install-17056-root-script-afghsec.sh: afghsec: root script finished successfully."
This is the resumed installation output after the root script execution:
At the end, the script reminds you to copy the generated zip file that contains the database acknowledge key back to the Hadoop cluster and then run ./jaguar databaseack to process the key.
bds-database-install: root script seem to have succeeded, continuing...
bds-database-install: cluster type : bda
bds-database-install: connection type : Infiniband
bds-database-install: cluster name : afghsec
bds-database-install: cluster version : cdh5
bds-database-install: dictionary cluster name : afghsec
bds-database-install: Cluster Hadoop version : 2.6.0-cdh5.12.1
bds-database-install: Hadoop client version : hadoop-2.6.0-cdh5.12.1
bds-database-install: Hive client version : hive-1.1.0-cdh5.12.1
bds-database-install: HBase client version : Not available on the cluster
bds-database-install: BDSQL version : 4.0.0
bds-database-install: BDSQL install date : Mon May 20 17:05:23 2019
bds-database-install: bd_cell version : bd_cell-12.2.1.1.0_LINUX.X64_190515.1-1.x86_64
bds-database-install: cp2hadoop kit version : orahivedp-5.0.0
bds-database-install: ohsh kit version : ohsh-5.0.0
bds-database-install: cell config dir : /etc/oracle/cell/network-config
bds-database-install: hybrid cell model : false
bds-database-install: cellinit.ora : missing
bds-database-install: cellaffinity.ora file : missing
bds-database-install: listener end point : EXTPROC1521
bds-database-install: installing JDK : jdk1.8.0_171
bds-database-install: creating cellinit backup files
bds-database-install: action : setup
bds-database-setup: validating <IP Address>/21 against network interfaces
bds-database-setup: found <IP Address>/21 at bondib0
bds-database-setup: ping to cluster nodes from <IP Address> (standard frame)
bds-database-setup: response ok from <IP Address>
bds-database-setup: response ok from <IP Address>
bds-database-setup: response ok from <IP Address>
bds-database-setup: ping to cluster nodes from <IP Address> (Jumbo frame)
bds-database-setup: response ok from <IP Address>
bds-database-setup: response ok from <IP Address>
bds-database-setup: response ok from <IP Address>
bds-database-setup: installing cluster configuration files
bds-database-setup: unpacking hadoop-2.6.0-cdh5.12.1.tar.gz
bds-database-setup: unpacking hadoop-2.6.0-nativelib-ol7.tar.gz
bds-database-setup: unpacking hive-1.1.0-cdh5.12.1.tar.gz
bds-database-setup: unpacking sentry-1.5.1-cdh5.12.1.tar.gz
bds-database-setup: creating database objects
catcon::set_log_file_base_path: ALL catcon-related output will be written to [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-directory-creation_catcon_20018.lst]
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-directory-creation*.log] files for output generated by scripts
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-directory-creation_*.lst] files for spool files, if any
catcon.pl: completed successfully
bds-database-setup: mta will be setup
bds-database-setup: creating /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/hs/admin/initbds_orcl_afghsec.ora
bds-database-setup: mta shutdown bds_orcl_afghsec
bds-database-setup: removing existing mta parameter settings
bds-database-setup: setting performance parameters for mta
bds-database-setup: setting listener_address to EXTPROC1521
bds-database-setup: starting extproc bds_orcl_afghsec
bds-database-setup: waiting upto 100 seconds for extproc to register with listener
................................ok
bds-database-setup: patching view LOADER_DIR_OBJS
catcon::set_log_file_base_path: ALL catcon-related output will be written to [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-patch-view_catcon_21550.lst]
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-patch-view*.log] files for output generated by scripts
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-patch-view_*.lst] files for spool files, if any
catcon.pl: completed successfully
bds-database-setup: creating mta dblinks
catcon::set_log_file_base_path: ALL catcon-related output will be written to [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-create-links_catcon_21746.lst]
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-create-links*.log] files for output generated by scripts
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-create-links_*.lst] files for spool files, if any
catcon.pl: completed successfully
catcon::set_log_file_base_path: ALL catcon-related output will be written to [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-alter-system_catcon_21936.lst]
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-alter-system*.log] files for output generated by scripts
catcon::set_log_file_base_path: catcon: See [/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install/bds-database-install-17056-alter-system_*.lst] files for spool files, if any
catcon.pl: completed successfully
bds-database-setup: installing cp2hadoop
bds-database-setup: installing ohsh
bds-database-setup: enabling TLS encrytion
bds-database-setup: reconfiguring database instance orcl
bds-database-setup: adding parameter _bigdata_external_table=true
bds-database-setup: adding parameter _dskm_single_instance=true
bds-database-setup: setup cell contiguration
bds-database-setup: creating cellinit.ora
bds-database-setup: cellaffinity.ora already disabled
bds-database-setup: verifying pdbs status
bds-database-setup: saving pdbs open states
bds-database-setup: restarting database instance
bds-database-setup: Object Store scripts templates have been created:
bds-database-setup: set_parameters_cdb.sql should be executed in CDB
bds-database-setup: allow_proxy_pdb.sql should be executed on required PDBs
/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/BDSJaguar-4.0.0/afghsec-3-scaj41bda18.us.oracle.com/set_parameters_cdb.sql
/u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/BDSJaguar-4.0.0/afghsec-3-scaj41bda18.us.oracle.com/allow_proxy_pdb.sql
bds-database-setup: please review and update them if requiered
bds-database-setup: activation zip file created: afghsec-3-scaj41bda18.us.oracle.com-scaj22bda08.us.oracle.com.zip
# /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/BDSJaguar-4.0.0/afghsec-3-scaj41bda18.us.oracle.com/afghsec-3-scaj41bda18.us.oracle.com-scaj22bda08.us.oracle.com.zip
bds-database-setup: copy this file to root@scaj41bda18.us.oracle.com:/opt/oracle/DM/databases/conf
bds-database-setup: and run:
./jaguar databaseack
bds-database-setup: configuration successful for cluster afghsec
bds-database-setup: log files in /u03/app/oracle/18.1.0/dbhome_orcl/shiphome/database/install
bds-database-setup: bds-database-install-17056-* files for more detailed info
bds-database-setup: full log in bds-database-install-17056-main-script.log
Copy the zip file generated by bds-database-install.sh back to the Hadoop cluster management server. This zip file contains the .ackkey file needed to complete the Database Authentication setup.
$ scp afghsec-3-<node name>.us.oracle.com-<node name>.us.oracle.com.zip root@<Hadoop node FQDN>:/opt/oracle/DM/databases/conf
root@<Hadoop node FQDN>'s password:
afghsec-3-<node name>.us.oracle.com-<node name>.us.oracle.com.zip 100% 789 0.8KB/s 00:00As root on the Hadoop cluster management server, run Jaguar’s Database Acknowledge operation. This completes the installation.
If your Jaguar configuration file is not named bds-config.json, then include the file name in the command, as in ./jaguar databaseack myjagconf.json.
[root@scaj41bda18 BDSJaguar]# ./jaguar databaseack
BigDataSQL: =======================================================================.
BigDataSQL: Oracle Big Data SQL 4.0.0.
BigDataSQL: Starting execution on: Mon May 20 17:15:06 2019.
BigDataSQL: in directory: /opt/oracle/BDSJaguar.
BigDataSQL: and hostname: <Hadoop node FQDN>>.
BigDataSQL: IP address: <IP Address>.
BigDataSQL: DB version support: 18.
BigDataSQL: Object Store support: Enabled.
BigDataSQL: API url: https://<Hadoop node FQDN>>:7183.
BigDataSQL: Waiting for management server to respond...
BigDataSQL: Configuration json file name: bds-config.json.
BigDataSQL: WARNING: If there are services in the stale state, they will be restarted !!!.
Cloudera Manager user: admin
Password:
BigDataSQL: Authentication succeeded.
BigDataSQL: API version: 17.
BigDataSQL: Cluster name : afghsec.
BigDataSQL: Cluster type : BDA.
BigDataSQL: Cluster version: CDH5.
BigDataSQL: full version: 5.12.1.
BigDataSQL: BDA version : 4.10.0-9.
BigDataSQL: Cluster health check..
BigDataSQL: Service: BIGDATASQL.
BigDataSQL: Verifying service configuration..
BigDataSQL: Connection type: Infiniband.
BigDataSQL: Initialization completed..
BigDataSQL: Starting deployment server..
BigDataSQL: Analyzing cluster ....
BigDataSQL: Checking for Big Data SQL requirements..
BigDataSQL: Database acknowledge..
BigDataSQL: Refreshing Big Data SQL database connections..
BigDataSQL: Starting bigdatasql service..
BigDataSQL: Setting database key for Big Data SQL..
BigDataSQL: Restarting stale services..
BigDataSQL: Restarting Cloudera Management service..
Big Data SQL setup finished !!!
run bdschecksw tool on database node for sanity verification
BigDataSQL: DATABASEACK workflow completed.
Example 2: Installation on an HDP Cluster
The installation on an HDP system is nearly the same as on CDH. As in the previous example, Oracle Big Data SQL will be configured to use Database Authorization feature. Therefore the --requestdb parameter is included in the Jaguar install command in order to generate a request key file.
# ./jaguar --requestdb orcl install
BigDataSQL: =======================================================================.
BigDataSQL: Oracle Big Data SQL 4.0.0.
BigDataSQL: Starting execution on: Wed May 22 16:00:40 2019.
BigDataSQL: in directory: /opt/oracle/BDSJaguar.
BigDataSQL: and hostname: <Hadoop node FQDN>.
BigDataSQL: IP address: 10.128.17.167.
BigDataSQL: DB version support: 18.
BigDataSQL: Object Store support: Enabled.
BigDataSQL: API url: http://<node.domain.com>:8080.
BigDataSQL: Waiting for management server to respond...
BigDataSQL: Configuration json file name: bds-config.json.
BigDataSQL: WARNING: If there are services in the stale state, they will be restarted !!!.
Ambari user: admin
Password:
BigDataSQL: Authentication succeeded.
BigDataSQL: API version: 1.
BigDataSQL: Cluster name : afghhdp2.
BigDataSQL: Cluster type : HDP.
BigDataSQL: Cluster version: HDP2.
BigDataSQL: full version: 2.6.5.1100-53.
BigDataSQL: Cluster health check..
BigDataSQL: Service: BIGDATASQL.
BigDataSQL: Verifying service configuration...
BigDataSQL: Connection type: Ethernet.
BigDataSQL: Initialization completed..
BigDataSQL: Copying Stack to Ambari..
BigDataSQL: Restarting management server...
BigDataSQL: Waiting for management server to respond...
BigDataSQL: Starting deployment server..
BigDataSQL: Analyzing cluster ....
BigDataSQL: Checking for Big Data SQL requirements..
BigDataSQL: Preparing installation..
BigDataSQL: Installing Big Data SQL agents..
BigDataSQL: Installing Big Data SQL servers..
BigDataSQL: Starting bigdatasql service.
BigDataSQL: reqkey file created: /opt/oracle/BDSJaguar/dbkeys/orcl.reqkey.
BigDataSQL: Restarting stale services..
BigDataSQL: Starting hdfs service.
BigDataSQL: Starting yarn service.
BigDataSQL: Starting mapreduce2 service.
BigDataSQL: Starting hbase service.
BigDataSQL: Starting bigdatasql service.
BigDataSQL: Big Data SQL task completed..
BigDataSQL: Creating database side bundle..
BigDataSQL: Creating bundle file.
BigDataSQL: Bundle file created: /opt/oracle/BDSJaguar/db-bundles/bds-4.0.0-db-afghhdp2-190522.1600.zip.
BigDataSQL: INSTALL workflow completed.After the HDP-side installation is complete, copy the database-side installation bundle and the key file to the database, unpack the bundle and execute the run file. Then, run bds-database-install.sh and copy the generated zip file containing the acknowledge key back to HDP and, finally, run ./jaguar databaseack. All of these steps are the same as shown in the Big Data Appliance example. The database side of the installation is the same, regardless of whether the Hadoop platform is CDH or HDP.