Getting Started with GGSA on Oracle Cloud

1.1 Overview

Oracle GoldenGate Stream Analytics (GGSA) on Oracle Cloud Marketplace enables customers to set up and run Oracle GGSA on Oracle Cloud. The cloud version provides the same functionality, scalability, security and support as the on-premise version. All sources and targets supported in the on-premise version are supported on cloud.

1.2 GGSA Architecture

Stream Analytics on Oracle Cloud Infrastructure Marketplace

This sample architecture diagram depicts a customer deployment of GGSA to use Autonomous Data Warehouse with Analytics Cloud as the visualization tool. GGSA supports many other target types and has its own charts and dashboards for real-time data viewing. GGSA on OCI Marketplace

Acquiring data

Stream Analytics can acquire data from any of the following on-premises and cloud-native data sources:

GoldenGate: Natively integrated with Oracle GoldenGate, Stream Analytics offers data replication for high-availability data environments, real-time data integration, and transactional change data capture.
Oracle Cloud Streaming: Ingest continuous, high-volume data streams that you can consume or process in real-time.
Kafka: A distributed streaming platform used for metrics collection and monitoring, log aggregation, and so on.
Java Message Service: Allows java-based applications to send, receive, and read distributed communications.

Processing data

With Stream Analytics on Oracle Cloud Infrastructure Marketplace, you can filter, correlate, and process events in real-time, in the cloud. Marketplace solutions are deployed to Oracle Cloud Infrastructure Compute instances within your regional subnet.

After Stream Analytics processes the data, you can:

store the data in Autonomous Data Warehouse, where it can be accessed by Analytics Cloud
configure the Notifications service to broadcast messages to other applications hosted on Oracle Cloud Infrastucture
send messages to OCI streaming service
send the processed data to the OCI Object store

Analyzing data

You can connect Analytics Cloud to Autonomous Data Warehouse so that users can analyze the data in visualizations, analyses, dashboards, and pixel-perfect reports. Learn more.

1.3 Resources

The GGSA marketplace image includes the following components installed:

GoldenGate Stream Analytics 19.1.0.0.6.1
Spark 3.0.2
Kafka 2.1.1
Mysql Enterprise 8.0.18
Java – JRE-8u281
Oracle GoldenGate for Big Data 19.1.0.0.8 (Classic Architecture)

Except for Java and Mysql Enterprise, all the other components are installed under the /u01/app directory structure.

1.4 Creating a GGSA Instance

You can create a GGSA instance and launch GoldenGate Stream Analytics on Oracle Cloud Marketplace using the Stack Listing, following the steps below:

Finding GGSA on Oracle Cloud Marketplace

Log in to the Oracle Cloud Marketplace.
From the Oracle Cloud Marketplace home page, use the search box under Applications and search for the keyword Stream Analytics .
From the search results, select GoldenGate Stream Analytics.

Deploying GGSA on Oracle Cloud Marketplace

From the Application page, select Get App.
Select OCI Region or Log in using your Single Sign-On credentials.
- OCI Region – Select the desired region and click Create Stack.
On the Oracle GoldenGate Stream Analytics page, provide the following information:
- Select Version - It provides a list of versions that are available
- Select Compartment - Specifies the compartment where the compute node will be built. It is generally the location that you have access to build the compute node.
- Terms of Use - This check box is selected by default. Oracle recommends to review the licenses before proceeding with the instance creation.
- Launch Stack - It launches the stack in the OCI environment.

Creating GGSA Resource Manager Stack

Fill in the required Stack information:
- Name - Name of the Stack. It has a default name and provides a date time stamp. You can edit this detail, if required.
- Description - Description that you provide while creating the Stack.
- Create In Compartment – It defaults to the compartment you have selected on the Oracle Marketplace page.
- Terraform Version - It defaults to 0.12x.
- Tags (optional) – Tags are a convenient way to assign a tracking mechanism but are not mandatory. You can assign a tag of your choice for easy tracking. You have to assign a tag for some environments for cost analysis purposes.
- Click Next.
Fill in the required details to configure variables.
- Name for New Resources -
  - Display Name – Display name used to identify all new OCI resources.
  - Host DNS Name – Name of the Domain Name Service for the new compute node.
    
    Note:
    Do not include a hyphen in the Host DNS Name, while provisioning a GGSA instance. You will see the following error if you include a hyphen: No connection to server. Check your network connection and refresh page. If the problem persist contact administrator.
- Network Settings -
  - Create New – Uncheck the check box Use Existing Network if you want to create new network infrastructure.
    1. VCN Network Compartment – Select the compartment in which you want to create the new VCN
    2. Subnet Network Compartment – Select the compartment in which you want to create the subnet.
    3. New VCN DNS Name – Name for the VCN
    4. New VCN CIDR Block – IP CIDR block for the VCN
    5. New Subnet DNS Name – Name for the new Subnet
    6. New Subnet CIDR Block – IP CIDR Block for the Subnet
  - Use existing Network – Check the Use existing Network check box, if you want to use existing network infrastructure.
    1. VCN Network Compartment – Select the compartment which has the existing VCN that you want to use
    2. Subnet Network Compartment – Select the compartment which has the existing Subnet that you want to use
    3. VCN – Select the VCN that you want to use.
    4. Subnet – Select the Subnet that you want to use.
    Note:
    
    If you are using an existing VCN and subnet, check the following if you want the instance to be accessible from public internet :
    - Your VCN must have an internet gateway
    - Your subnet should have security rules to allow ingress HTTP traffic for the following ports : 443, 7801, 7811-7830
      
      443 – has ngnix running which fronts the OSA UI and spark console
      
      7801 – is the port for the GG Manager when we install GoldenGate Big Data (GGBD)
      
      7811-7830 –When we start GG Change data from OSA UI, it creates a GG Distribution path and starts the GG Replicat process. These are the ports used by Oracle GG processes to bind for communication with a remote Oracle GG process.
    - Your subnet should be a public subnet and should have route rule with target type set to Internet Gateway, and should use the Gateway of the VCN.
- Instance Settings –
  1. Availability Domain – It specifies the availability domain for the newly created GGSA Instance. It must match the Availability Domain of Subnet that you have selected in the Use Existing Network settings.
  2. Compute Shape – Shape of new compute instance. Supported shapes are VM.Standard2.4, VM.Standard2.8, VM.Standard2.16 and VM.Standard2.24
  3. Assign Public IP – This option indicates if the newly created VM should have a public IP address. This option is selected by default. If you clear this check box, no public IP address will be assigned preventing public access to the compute node.
    It is important to provide your RSA public key for accessing the instance using ssh. SSH is the only way to retrieve the passwords for GoldenGate Stream Analytics and MySQL Metadata store. RSA public key will start with ssh-rsa.
  4. Custom Volume Sizes – Select this check box to customize the size of the new block storage volumes that are built for the compute node.
    1. Boot Volume – 50 GB
    2. Kafka Logs – 50 GB
    3. Mysql Datafile – 50 GB
    4. GG4BD (Golden Gate For Big Data) Datadir – 50GB
    Note:
    GGSA Resource Manager Stack will create 3 additional block volumes apart from the boot volume attached to the instance. The naming conventions for these block volume is :
    
    <Display Name> (Kafka Logs)
    
    <Display Name> (Mysql Datafile)
    
    <Display Name> (GG4BD Datadir)
    
    The Display Name is what you specified earlier.
    You can configure backup for these block volumes by subscribing to the OCI Backup Policy. To apply the Backup Policy, to the Block Volume, go to Block Storage -> Block Volume and select the block volume for which you want to configure the backup. Click Apply.
- Use Existing Block Volume – Select this option if you already have block volumes from your previous install that has data that you want to reuse.
  1. Kafka Logs Block Volume – Provide the OCID of the block volume which was used for Kafka logs in your previous setup.
  2. MySql Datafile Block Volume – Provide the OCID of the block volume that was used for Mysql datafile storage in your previous setup
  3. GG4BD Datadir - Provide the OCID of the block volume that was used for Mysql datafile storage in your previous setup
- Shell Access
  - SSH Public Key – Public Key for allowing SSH access as the opc user. Enter the key and click Next.
Review -
- On the Review page, review the information you provided and then click Create. The OCI Resource Manager Stack is created, and Terraform Apply action is initiated to create the OCI resources. You can see the created stack in the list of stacks in the Resource Manager - > Stacks section. You can manage the stack here.

Using Terraform Actions on Stack

Once the Stack is created, you can apply Terraform Actions on it to plan, create and destroy the OCI resources.

Terraform Actions on Stack

Plan – Terraform Plan performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state.
Apply – Use the Terraform Apply action to create the OCI resources required to reach the required state as per the user selection in the stack creation.
Destroy – Use the Terraform Destroy action to destroy all the OCI resources created by the Stack during the Apply action.

Note:
All the resources created by the Apply action will be destroyed if you use the Destroy action, so be careful when using this action.

1.5 Validating a GGSA Instance

Checking the status of the GGSA Services

To check the status of the GGSA Services, connect to the newly created instance using SSH and run the command ggsa-servcies.

To connect to the instance, use the private key corresponding to the public key, provided at the time of stack creation.

You can find the Instance IP in the logs of the Apply job, or from Outputs under Resources on the Job Details page: ggesp_public_ip.

The following example illustrates how you can connect to the OSA compute node:ssh -i <PRIVATE_KEY_FILE_PATH> opc@INSTANCE_IP

You should see the following GGSAservices active and running:

Check the GGSA Service Status

Note:

If you see the OSA service to be dead or failed, wait for about 10 minutes for the services to start.

Logging in to the GGSA UI

Open the README.txt file in the home directory, and copy your OSA Admin password as shown in the line.
###### OSA UI

Your osaadmin password to login to OSA UI is XXXXXXXXX
From your Chrome browser, type https://<IP_of_Instance> or https://<IP_of_Instance/osa>, to display the login page.
Login using osaadmin as username and the password copied from README.txt to display the Home page.

Note:

As mentioned in the previous section, wait for all the services to start, else the README file will have only placeholder passwords and not the actual passwords.

1.6 Starting and Stopping GGSA Services

All the GoldenGate Stream Analytics services listed above are registered as system services and are automatically started when an instance is successfully provisioned, and/ or the system is rebooted. Any published pipelines are also automatically restarted in case of a system reboot.

To start, stop, or restart the services, use the following commands from an ssh terminal:

sudo systemctl start osa /** starts OSA **/

sudo systemctl stop osa /** stops OSA **/

sudo systemctl restart osa /** restarts OSA **/

Start and Stop GGSA Services

1.7 Securing your GGSA Instance

Connect to your newly created instance using ssh, and open the README.txt in the home directory. This file includes the passwords for MySQL root user and the GGSA Admin user. Additionally it includes the passwords for GGSA DB schema and a OSA Demo Database in MySQL, which you have the option to change. To change the password for GGSA DB schema, update the password in /u01/app/osa/osa-base/etc/jetty-osa-datasource.xml and restart GGSA.

Change MySQL Root/Admin Password

Note down or copy the MySQL admin password from README.txt file in home directory. You can change the MySQL root/admin password as shown in the commands below.

Change MySQL Root Password

Note:

Similarly you can change OSA_DEMO user password with mysql -u OSA_DEMO -p.

Change GGSA Admin Password

Note down or copy the GGSA admin password to clipboard from README.txt file in home directory.

You can change the Admin password in the User Management tab of the System Settings screen on the UI.

Setup Spark Admin User and Password for Spark console

You can set the sparkadmin user and password in the Environment tab of the System Settings screen on the UI. You will need this to access the Spark Console.