How to Install GoldenGate Stream Analytics?

1.1 Planning Your Installation

To plan the installation of GoldenGate Stream Analytics (GGSA) 19.1.0.0.6 efficiently, ensure that you have the required hardware and software. You should also perform the prerequisite procedures before starting the installation process.

You can use the information in the certification matrix before installing GoldenGate Stream Analytics 19.1.0.0.6. The certification matrix provides you useful links to support pages, supported software, and system requirements in general. The following software is required for operation of GGSA:

Oracle JDK 8 Update 131 and higher versions
Repository Database
- Oracle Database versions 12.2.0.1 or higher, 12.1.0.1 or higher, and 11.2.0.4 or higher
- Else, you can use MySQL version 5.6 or 5.7
A running Hadoop cluster with version 2.7 or higher
A running Kafka cluster with version 0.10.2 or higher
Locally installed Spark Libraries with version 2.2.1 - 2.4.3, built for Hadoop 2.7. OSA does not package the Spark client libraries, so you will also need locally installed Spark with version 2.4.3 built for Hadoop 2.7.

Note:
Install Spark and JDK in the same node on which you plan to install Oracle Stream Analytics. See Installing GoldenGate Stream Analytics.
Google Chrome browser with version 6.0 or higher

1.2 Installing GoldenGate Stream Analytics

After you have reviewed the above software prerequisites, please follow the steps below to install GoldenGate Stream Analytics 19.1.0.0.6:

Create a directory, for example, spark-downloads, and download Apache Spark into the newly created folder and as specified by versions below:
- Spark release: 2.4.3
- Package type: Pre-built for Apache Hadoop 2.7 and later
- Download Spark: spark-2.4.3-bin-hadoop2.7.tgz
Extract the Spark archive to a local directory.
You can see a subfolder, spark-2.4.3-bin-hadoop2.7.
Create a new directory, for example, OSA-19 and download OSA-19.1.0.0.6.zip from Oracle eDelivery and extract it into the newly created folder.
You can find the OSA-19.1.0.0.6-README.txt file in the OSA-19.1.0.0.6 zip file.
Extract the downloaded file. You should now see a subfolder OSA-19.1.0.0.6.
Review the file OSA-19.1.0.0.6-README.txt in the OSA-19 folder.
Set the environment variables:
1. Set the SPARK_HOME environment variable in the OSA-19.1.0.0.6/osa-base/etc/osa-env.sh file to point to the directory where you have extracted the Spark archive. For example:
  SPARK_HOME=/products/spark-downloads/spark-2.4.3-bin-hadoop2.7
Set the JDK_HOME environment variable in the OSA-19.1.0.0.6/osa-base/etc/osa-env.sh file to point to the directory where you have extracted the JDK archive. For example:
JDK_HOME=/products/java-downloads/jdk1.8.0_131

1.3 Configuring the Metadata Store

Please follow steps below for configuring your metadata store.

Configure your data source in OSA-19.1.0.0.6/osa-base/etc/jetty-osa-datasource.xml as per instructions below. This step is essential for creating OSA’s database schema. The OSA database user referred to in the document will be created by the installation process.

Uncomment and edit one of the two Data source configurations, either for Oracle Database or MySQL depending on the database you want to use as metadata store. The uncommented fragment for Oracle database is shown below: a.

<New id="osads" class="org.eclipse.jetty.plus.jndi.Resource">               
          <Arg>             
          <Ref refid="wac"/>
          </Arg>        
              <Arg>jdbc/OSADataSource</Arg>         
            <Arg>             
             <New class="oracle.jdbc.pool.OracleDataSource">
                <Set name="URL">jdbc:oracle:thin:@myhost.example.com:1521:OSADB</Set>
                <Set name="User">OSA_USER</Set>
                <Set name="Password">
                  <Call class="org.eclipse.jetty.util.security.Password" name="deobfuscate">                       
                    <Arg> OBF:OBFUSCATED_PASSWORD</Arg>                   
                  </Call>               
                </Set>                
                <Set name="connectionCachingEnabled">true</Set>                
                <Set name="connectionCacheProperties">  
                 <New class="java.util.Properties">   
                 <Call name="setProperty"><Arg>MinLimit</Arg><Arg>1</Arg></Call> 
                 <Call name="setProperty"><Arg>MaxLimit</Arg><Arg>15</Arg></Call>   
                 <Call name="setProperty"><Arg>InitialLimit</Arg><Arg>1</Arg></Call> 
                 </New>               
               </Set>             
             </New>         
            </Arg>
        </New>

Decide on an OSA schema username and a plain-text password. For illustration, say osa as schema user name and alphago as password.
Change directory to top-level folder OSA-19.1.0.0.6 and execute the following command:java -cp ./lib/ jetty-util-9.4.17.v20190418.jar org.eclipse.jetty.util.security.Password osa <your password>

For example, java -cp ./lib/ jetty-util-9.4.17.v20190418.jar org.eclipse.jetty.util.security.Password osa alphago

You should see results like below on console:
2019-06-18 14:14:45.114:INFO::main: Logging initialized @1168ms to org.eclipse.jetty.util.log.StdErrLogalphago
OBF:<obfuscated password>

MD5:34d0a556209df571d311b3f41c8200f3

CRYPT:osX/8jafUvLwA
Note down the obfuscated password string that is displayed (shown in bold), by copying it to clipboard or notepad.
Change database host, port, SID, osa schema user name and osa schema password fields marked in bold in the code in Step 2a.
Example - jdbc:oracle:thin:@myhost.example.com:1521:ORCL

SAMPLE JETTY-OSA-DATASOURCE.XML

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">

<!-- =============================================================== -->
<!-- Configure jdbc/OSADataSource data source                        -->
<!-- =============================================================== -->
<Configure id="Server" class="org.eclipse.jetty.server.Server">

    <!-- SAMPLE OSA DATASOURCE CONFIGURATION FOR ORACLE-->
    <New id="osads" class="org.eclipse.jetty.plus.jndi.Resource">
         <Arg>
             <Ref refid="wac"/>
         </Arg>
         <Arg>jdbc/OSADataSource</Arg>
         <Arg>
             <New class="oracle.jdbc.pool.OracleDataSource">
                 <Set name="URL">jdbc:oracle:thin:@myhost.example.com:1521:OSADB</Set>
                 <Set name="User">osa_prod</Set>
                 <Set name="Password">
                    <Call class="org.eclipse.jetty.util.security.Password" name="deobfuscate">
                        <Arg>OBF:1ggz1j1u1k8q1leq1v2h1w8v1v1x1lcs1k5g1iz01gez</Arg>
                    </Call>
                </Set>
                <Set name="connectionCachingEnabled">true</Set>
                <Set name="connectionCacheProperties">
                    <New class="java.util.Properties">
                        <Call name="setProperty"><Arg>MinLimit</Arg><Arg>1</Arg></Call>
                        <Call name="setProperty"><Arg>MaxLimit</Arg><Arg>15</Arg></Call>
                        <Call name="setProperty"><Arg>InitialLimit</Arg><Arg>1</Arg></Call>
                    </New>
                </Set>
             </New>
         </Arg>
     </New>


    <!-- SAMPLE OSA DATASOURCE CONFIGURATION FOR ADW-->
    <!--
    <New id="osads" class="org.eclipse.jetty.plus.jndi.Resource">
         <Arg>
             <Ref refid="wac"/>
         </Arg>
         <Arg>jdbc/OSADataSource</Arg>
         <Arg>
             <New class="oracle.jdbc.pool.OracleDataSource" type="adw">
                 <Set name="URL">jdbc:oracle:thin:@oracletestdb_high?TNS_ADMIN=/scratch/oracletest/Wallet_oracletestdb</Set>
                 <Set name="User">{OSA_USER}</Set>
                 <Set name="Password">
                    <Call class="org.eclipse.jetty.util.security.Password" name="deobfuscate">
                        <Arg>{OBF:OBFUSCATE_PASSWORD}</Arg>
                    </Call>
                </Set>
                <Set name="connectionCachingEnabled">true</Set>
                <Set name="connectionCacheProperties">
                    <New class="java.util.Properties">
                        <Call name="setProperty"><Arg>MinLimit</Arg><Arg>1</Arg></Call>
                        <Call name="setProperty"><Arg>MaxLimit</Arg><Arg>15</Arg></Call>
                        <Call name="setProperty"><Arg>InitialLimit</Arg><Arg>1</Arg></Call>
                    </New>
                </Set>
             </New>
         </Arg>
     </New>
     -->
    <!-- SAMPLE OSA DATASOURCE CONFIGURATION FOR MYSQL-->
    <!--

    <New id="osads" class="org.eclipse.jetty.plus.jndi.Resource">
        <Arg>
            <Ref refid="wac"/>
        </Arg>
        <Arg>jdbc/OSADataSource</Arg>
        <Arg>
            <New class="com.mysql.cj.jdbc.MysqlConnectionPoolDataSource">
                <Set name="URL">jdbc:mysql://examplehost.com:3306/OSADB</Set>
                <Set name="User">{OSA_USER}</Set>
                <Set name="Password">
                    <Call class="org.eclipse.jetty.util.security.Password" name="deobfuscate">
                        <Arg>{OBF:OBFUSCATE_PASSWORD}</Arg>
                    </Call>
                </Set>
            </New>
        </Arg>
    </New>
-->

</Configure>

Note:

Do not use a hyphen in the OSA metadata username, in the jetty-osa-datasource.xml

.

1.3.1 Configuring ATP/ADW as Metadata Store

GoldenGate Stream Analytics creates the metadata schema, as part of initial configuration of the system, using the script: ${OSA_HOME}/osa-base/bin/configure.sh dbroot=<sys user of database> dbroot_password=<sys user password of the database>

However, before running the above script, you must configure the datasource in the datasource configuration file at${OSA_HOME}/osa-base/etc/jetty-osa-datasource.xml.

To configure ATP/ADW as metadata store, first comment the Oracle and MYSQL sections, while uncommenting the ADW/APT section in jetty-osa-datasource.xml file.

Below is the template for the datasource configuration for ATP/ADW database:

jetty-osa-datasource.xml

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <New id="osads" class="org.eclipse.jetty.plus.jndi.Resource">
         <Arg>
             <Ref refid="wac"/>
         </Arg>
         <Arg>jdbc/OSADataSource</Arg>
         <Arg>
             <New class="oracle.jdbc.pool.OracleDataSource" type="adw">
                 <Set name="URL">jdbc:oracle:thin:@{service_name}?TNS_ADMIN={wallet_absolute_path}</Set>
                 <Set name="User">{osa_db_user}</Set>
                 <Set name="Password">
                    <Call class="org.eclipse.jetty.util.security.Password" name="deobfuscate">
                        <Arg>{obfuscated_password}</Arg>
                    </Call>
                </Set>
                <Set name="connectionCachingEnabled">true</Set>
                <Set name="connectionCacheProperties">
                    <New class="java.util.Properties">
                        <Call name="setProperty"><Arg>MinLimit</Arg><Arg>1</Arg></Call>
                        <Call name="setProperty"><Arg>MaxLimit</Arg><Arg>15</Arg></Call>
                        <Call name="setProperty"><Arg>InitialLimit</Arg><Arg>1</Arg></Call>
                    </New>
                </Set>
             </New>
         </Arg>
     </New>
</Configure>

Note:

In the above template, replace the variables in {} as below:

{service_name} - one of the service names listed in the tnsnames.ora file inside the wallet
{wallet_absolute_path} - the absolute path of wallet folder on the machine where OSA is installed
{osa_db_user} - the username to create the osa metadata. This username and schema will be created by the 'dbroot' user provided in above script.
{obfuscated_password} - the Obfuscated password for {osa_db_user}

1.4 Initializing Metadata Store

This topic applies only to Oracle user-managed services.

After installing GGSA, you need to configure the metadata store with the database admin credential details and the version of GGSA as required.

To initialize the metadata store, you need database admin credentials with sysdba privileges:

Change directory to OSA-19.1.0.0.6/osa-base/bin.
Execute the following command:./start-osa.sh dbroot=<db sys user> dbroot_password=<db sys password> For example, ./start-osa.sh dbroot=AlphaUser dbroot_password=AlphaPassword

Following console messages will be displayed indicating the OSA schema was created and prompting for password for osaadmin user.

The following console messages indicates that the GGSA schema is created and the metadata store is successfully initialized:

JAVA_HOME: JAVA_HOME: /<jdk install path>/jdk1.8.0_121.
         SPARK_HOME: /<Spark install path>/spark-2.4.3-bin-hadoop2.7
         The RELEASE file exists:
         /<Spark install path>/spark-2.4.3-bin-hadoop2.7/RELEASE
         SPARK_VERSION: Spark 2.4.3 built for Hadoop 2.7.3
         2019-06-23 08:48:51.444:INFO::main:
         Logging initialized @305ms to org.eclipse.jetty.util.log.StdErrLog
         OSA DB user created:osa
         The OSA application administrative user with the predefined name "osaadmin" is going to be created
         You have not specified a password
         for the "osaadmin" user on the command line. Please enter it below.

Enter password:
Reenter password:
Run ./stop-osa.sh to complete schema creation and metadata initialization.
If you don’t see the above messages, check the OSA-19.1.0.0.6/osa-base/logs folder to identify the cause and potential solution.

Note:
If you do not have the database admin credentials, ask your database administrator to create a Oracle Stream Analytics database user by using the SQL scripts available in the OSA-19.1.0.0.6/osa-base/sql folder. The Oracle Stream Analytics database username must match the one configured in jetty-osa-datasource.xml.

1.5 Jetty Properties File

Use the jetty properties available at OSA-19.1.0.0.6/osa-base/etc/jetty.properties, to modify certain security features.

Note:

It is recommended that you configure these properties at the installation stage, to avoid restarting your server, if configured at a later stage.

Following are the available properties:

jetty.session.timeout
You can set the timeout for OSA web session. This sets the timeout for OSA web session. By default the timeout is set to 30 minutes. The value can be changed to any integer greater than 1.
host.headers.whitelist
You can restrict the x-forwarded-host header values to the values defined with this property.

Example: host.headers.whitelist= www.oracle.com, www.microsoft.com, localhost:9080

Here the value of the host header can be only of these three domains listed. Commenting out this property with a # will allow all values for the header.

Note:
If you do not specify explicitly the host header in your request, the default value is host-server:port, where the OSA jetty server is running. Hence you must specify the port number along with the server address.
xforwarded.host.headers.whitelist
You can restrict the x-forwarded-host header values to the values defined with this property.

Example: xforwarded.host.headers.whitelist= www.oracle.com, www.microsoft.com, localhost

Here the value of the x-forwarded-host header can be only of these three domains listed. Commenting out this property with a # will allow all values for the header. If no domain is entered, that is, if the value of the property is empty, then this header is not supported.
response.headers.list
A comma separated list of response headers, which will be sent along with response for every request.

Example: response.headers.list="x-frame-options: sameorigin, X-Content-Type-Options: nosniff"
By default the above 2 response headers are set.
- x-frame-options: sameorigin will prevent clickjack attacking.
- X-Content-Type-Options: nosniff will prevent sniffing of the response content by the browsers.

1.6 Adjusting Jetty Threadpool

Edit OSA-19.1.0.0.6/etc/jetty-threadpool.xml to change minimum and maximum thread configuration to 100 and 2000 respectively. Sample shown below.

<New id="threadPool"
    class="org.eclipse.jetty.util.thread.QueuedThreadPool">
 <Set name="minThreads" type="int"><Property name="jetty.threadPool.minThreads"
      deprecated="threads.min" default="100"/></Set>    
<Set name="maxThreads" type="int"><Property name="jetty.threadPool.maxThreads"
      deprecated="threads.max" default="2000"/></Set>    
<Set name="reservedThreads" type="int"><Property
      name="jetty.threadPool.reservedThreads" default="-1"/></Set>   
 <Set name="idleTimeout" type="int"><Property
      name="jetty.threadPool.idleTimeout" deprecated="threads.timeout"
      default="60000"/></Set>    
<Set name="detailedDump" type="boolean"><Property
      name="jetty.threadPool.detailedDump" default="false"/></Set>
</New>
</Configure>

1.7 Integrating Stream Analytics with Oracle GoldenGate

Follow the below steps to integrate Oracle Goldengate with Stream Analytics:

Download and install Oracle GoldenGate Big Data. For a compatible version of Oracle GoldenGate Big Data, see the latest certification matrix.

Note:
Install Oracle GoldenGate Big Data on the same machine and with the same user as OSA.
Set the following environment variables:
- KAFKA_HOME – set this variable to the path where Kafka is installed. Example :export KAFKA_HOME=/u01/app/kafka.
- LD_LIBRARY_PATH – set this variable to the directory path that contains JVM shared library.
  Example: export LD_LIBRARY_PATH=/u01/app/java/jre/lib/amd64/server:$LD_LIBRARY_PATH
- GGBD_HOME – set this variable to the path where Goldengate for Bigdata is installed.
  Example: export GGBD_HOME=/u01/app/OGG_BigData_Linux_x64_19.1.0.0.0
Start the manager process on port 7801.

For installation steps, see Installing GoldenGate for Big Data.

1.8 Maven Setting for GoldenGate Big Data Handlers

Maven is required to download third-party client libraries for the GGBD handlers to work.

1.8.1 Set the Maven Home Path

To configure maven home:

Update the OSA-19.1.0.0.6/osa-base/bin/configure-osa.sh with the correct M2_HOME path, as below:

Change the path from

OSA_HOME="$( cd "$(dirname "../../")" >/dev/null 2>&1 ; pwd -P )"

to

OSA_HOME="$( cd "$(dirname "../../../")" >/dev/null 2>&1 ; pwd -P )"

Note:

Update the maven home path before initialization of the metadata store, or you will have to restart GGSA after this update.

1.8.2 Configure Maven Proxy Settings

If your GGSA installation is behind proxy, to use the GGBD handlers, you have to configure the settings.xml that comes with the Maven distribution.

Update the <OSA_INSTALLATION_PATH>/apache-maven-3.6.3/conf/settings.xml with the correct proxy entries in the <proxies> </proxies> section, as shown below:

<proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <username>proxyuser</username>
      <password>proxypass</password>
      <host>proxy.host.net</host>
      <port>80</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
    </proxy>

Note:

Username and password field is required if the proxy is protected.

Note:

Update the settings.xml before initialization of the metadata store, or you will have to restart GGSA after this update.