7.6 Realtime Parquet Ingestion into OCI Object Storage with Oracle GoldenGate for Distributed Applications and Analytics

Overview

This Quickstart covers a step-by-step process showing how to ingest parquet files into OCI Object Storage buckets in real-time with GoldenGate for Distributed Applications and Analytics (GG for DAA).

OCI Object Storage is a scalable, high-performance storage service provided by Oracle Cloud Infrastructure (OCI). It enables users to store and manage large amounts of unstructured data, such as images, videos, log files, backups, and other types of files.

GG for DAA OCI Object Storage handler works in conjunction with File Writer Handler and Parquet Handler (if parquet is required). File Writer Handler produces files locally, optionally Parquet Handler converts to parquet format and OCI Object Storage Handler loads into OCI Object Storage buckets.

• Prerequisites
  
• Install Dependency Files
  
• Configure Credentials for Oracle Cloud Infrastructure
  
• Create a Replicat in Oracle GoldenGate for Distributed Applications and Analytics
  

7.6.1 Prerequisites

To successfully complete this Quickstart, you must have the following:

• OCI Object Storage access

In this Quickstart, a sample trail file (named tr), which is shipped with GG for DAA is used. The sample trail file is located at GG_HOME/opt/AdapterExamples/trail/ in your GG for DAA instance.

7.6.2 Install Dependency Files

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) uses client libraries in the replication process and these libraries need to be downloaded before setting up the replication process. You can use dependency downloader to download the client libraries. Dependency Downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.

GG for DAA uses a 3-step process to ingest parquet into OCI Object Storage buckets:

• Generating local files from trail files
• Converting local files to Parquet format
• Loading files into into OCI Object Storage buckets

For generating local parquet files with GG for DAA, replicat uses File Writer Handler and Parquet Handler.To load the parquet files into OCI Object Storage, GG for DAA uses OCI Event Handler in conjunction with File Writer and Parquet Event Handler.

GG for DAA uses 3 different set of client libraries to create parquet files and loading into OCI Object Storage:

1. In your GG for DAA VM, go to Dependency Downloader utility. It is located at GG_HOME/opt/DependencyDownloader/
2. Execute parquet.sh and hadoop.sh with and oracle_oci.sh with the required versions.

   Figure 7-35 Execute parquet.sh and hadoop.sh with and oracle_oci.sh with the required versions

   
3. 3 directories are created in GG_HOME/opt/DependencyDownloader/dependencies. Note the directories. For example:
   • /u01/app/ogg/opt/DependencyDownloader/dependencies/oracle_oci_3.2.0/*
   • /u01/app/ogg/opt/DependencyDownloader/dependencies/hadoop_3.3.0/*
   • /u01/app/ogg/opt/DependencyDownloader/dependencies/parquet_1.12.3/*

7.6.3 Configure Credentials for Oracle Cloud Infrastructure

You need to create a configuration file to authenticate into OCI. The ideal configuration file include user, fingerprint and key_file, tenancy, and region with their respective values. The default configuration file name and location is ~/.oci/config. For more information, see Required Keys and OCIDs documentation.

Sample Config File

[DEFAULT]
user=<your_user_ocid>
fingerprint=<your_fingerprint>
key_file=~/.oci/oci_api_key.pem #path-to_your_key_file
tenancy=<your_tenancy_ocid>

7.6.4 Create a Replicat in Oracle GoldenGate for Distributed Applications and Analytics

To create a replicat in Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA):

1. In the GG for DAA UI, in the Administration Service tab, click the + sign to add a replicat.

   Figure 7-36 Click the Administration Service tab

   
2. Select the Classic Replicat Replicat Type and click Next. There are two different Replicat types available: Classic and Coordinated. Classic Replicat is a single threaded process whereas Coordinated Replicat is a multithreaded one that applies transactions in parallel.

   Figure 7-37 Add Replicat

   
3. Enter the Replicat information, and click Next:
   1. Replicat Trail: Name of the required trail file. For sample trail, provide tr.
   2. Subdirectory: Enter GG_HOME/opt/AdapterExamples/trail/ if using the sample trail.
   3. Target: OCI Object Storage

   Figure 7-38 Replicat Options

   
4. Leave Managed Options as is and click Next.

   Figure 7-39 Managed Options

   
5. Enter Parameter File details and click Next. In the Parameter File, you can either specify source to target mapping or leave it as-is with a wildcard selection. If Coordinated Replicat is selected as the Replicat Type, then an additional parameter needs to be provided: TARGETDB LIBFILE libggjava.so SET property=<ggbd-deployment_home>/etc/conf/ogg/your_replicat_name.properties

   Figure 7-40 Parameter File

   
6. In the Properties File, remove all the pre-configured properties; but not the first row marked with the replicat name (# Properties file for Replicat <replicat_name>). Copy and paste below property list into properties file, update the properties marked as #TODO and click Create and Run.

   #The File Writer Handler – no need to change
   gg.handlerlist=filewriter
   gg.handler.filewriter.type=filewriter
   gg.handler.filewriter.mode=op
   gg.handler.filewriter.pathMappingTemplate=./dirout
   gg.handler.filewriter.stateFileDirectory=./dirsta
   gg.handler.filewriter.fileRollInterval=7m
   gg.handler.filewriter.inactivityRollInterval=5s
   gg.handler.filewriter.fileWriteActiveSuffix=.tmp
   gg.handler.filewriter.finalizeAction=delete
   
   
   
   ### Avro OCF – no need to change 
   gg.handler.filewriter.format=avro_row_ocf
   gg.handler.filewriter.fileNameMappingTemplate=${groupName}_${fullyQualifiedTableName}_${currentTimestamp}.avro
   gg.handler.filewriter.format.pkUpdateHandling=delete-insert
   gg.handler.filewriter.format.metaColumnsTemplate=${optype},${position}
   gg.handler.filewriter.format.iso8601Format=false
   gg.handler.filewriter.partitionByTable=true
   gg.handler.filewriter.rollOnShutdown=true
   
   
   #The Parquet Event Handler – no need to change
   gg.handler.filewriter.eventHandler=parquet
   gg.eventhandler.parquet.type=parquet
   gg.eventhandler.parquet.pathMappingTemplate=./dirparquet
   gg.eventhandler.parquet.fileNameMappingTemplate=${groupName}_${fullyQualifiedTableName}_${currentTimestamp}.parquet
   gg.eventhandler.parquet.writeToHDFS=false
   gg.eventhandler.parquet.finalizeAction=delete
   
   
   #TODO Select OCI Event Handler  – no need to change
   gg.eventhandler.parquet.eventHandler=oci
   
   #TODO Set OCI Event Handler - please update as needed
   gg.eventhandler.oci.type=oci
   gg.eventhandler.oci.region=<your_bucket_region>
   gg.eventhandler.oci.compartmentID=<your_compartment_ocid>
   gg.eventhandler.oci.bucketMappingTemplate=<your_bucket_name>
   gg.eventhandler.oci.pathMappingTemplate=${schemaName}
   gg.eventhandler.oci.fileNameMappingTemplate=${tableName}_${currentTimestamp}.parquet
   gg.eventhandler.oci.finalizeAction=NONE
   gg.eventhandler.oci.configFilePath=path_to_oci_config_file_from_step2
   #TODO: Edit to include the OCI Java SDK.
   
   
   #TODO Set the classpath to the paths you noted in step1
   gg.classpath=path_to/oracle_oci_3.2.0/*:path_to/Hadoop_3.3.0/*:path_to/parquet_1.12.3/*
   jvm.bootoptions=-Xmx512m -Xms32m
   

7. If replicat starts successfully, then it is in running state. You can go to Replicats/Statistics to see the replication statistics.

   Figure 7-41 Replication Statistics

   
8. Go to the OCI console and check the bucket.

   Figure 7-42 OCI Console

   

Note:

• If target OCI Object Storage bucket does not exist, it will be auto created by GG for DAA. You can use Template Keywords to dynamically assign the container names.
• OCI Object Storage Event Handler can be configured for proxy server. For more information, see OCI Object Storage Event Handler.
• You can use different properties to control the behaviour of file writing. You can set file sizes, inactivity periods and more. You can get more details in the File Writer blog post.