7.7 Realtime Message Streaming to AWS Kinesis

This topic covers a step-by-step process that shows how to ingest parquet files into AWS Kinesis in real-time with Oracle GoldenGate for Big Data.

• Install Dependency Files
  
• Create a Replicat in Oracle GoldenGate for Big Data
  

7.7.1 Install Dependency Files

Oracle GoldenGate for Big Data uses client libraries in the replication process. You need to download these libraries by using the Dependency Downloader utility available in Oracle GoldenGate for Big Data before setting up the replication process. Dependency downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.

Oracle GoldenGate for Big Data uses AWS Kinesis Java SDK to push data to Amazon Kinesis.

Note:

Oracle GoldenGate for Big Data does not ship with the AWS Kinesis Java SDK.

To install the required dependency files:

1. Go to installation location of Dependency Downloader: GG_HOME/opt/DependencyDownloader/.
2. Execute aws.sh with the required version.

   Figure 7-41 Executing aws.sh with the required version

   

   A directory is created in GG_HOME/opt/DependencyDownloader/dependencies. For example,/u01/app/ogg/opt/DependencyDownloader/dependencies/aws_sdk_1.12.30.

7.7.2 Create a Replicat in Oracle GoldenGate for Big Data

To create a replicat in Oracle GoldenGate for Big Data:

1. In the Oracle GoldenGate for Big Data UI, in the Administration Service tab, click the + sign to add a replicat.

   Figure 7-42 Click + in the Administration Service tab.

   
2. Select the Classic Replicat Replicat Type and click Next.

   Figure 7-43 Select the Replicat Type and click Next.

   
3. Enter the basic information, and click Next:
   1. Process Name: Name of the Replicat
   2. Trail Name: Name of the required trail file
   3. Target: AWS Kinesis

      Figure 7-44 Enter the Basic information and click Next.

      
4. Enter Parameter File details and click Next. In the Parameter File, you can either specify source to target mapping or leave it as is with a wildcard selection.

   Figure 7-45 Provide Parameter File details and click Next.

   
5. Enter the required properties:

   # Properties file for Replicat Kinesis
   #Kinesis Streams Handler Template
   gg.handlerlist=kinesis
   gg.handler.kinesis.type=kinesis_streams
   gg.handler.kinesis.mode=op
   gg.handler.kinesis.format=json
   #TODO: Set the region name for the connection.
   gg.handler.kinesis.region=<name_of_aws_region>
   #TODO: Set the template to resolve the Kinesis stream name.
   gg.handler.kinesis.streamMappingTemplate=<kinesis_stream_name>
   #TODO: Set the template to resolve the message key
   gg.handler.kinesis.partitionMappingTemplate=${primaryKeys}
   #TODO: Set the access key and secret key credentials.  If unset it will fall back to the AWS default credentials provider chain.
   gg.handler.kinesis.accessKeyId=<access_ley>
   gg.handler.kinesis.secretKey=<secret>
   #TODO: Set the path to the AWS SDK. (refer to step 1 of this document)
   gg.classpath=/u01/app/ogg/opt/DependencyDownloader/dependencies/aws_sdk_1.12.30/*
   jvm.bootoptions=-Xmx512m -Xms32m  
   

   Note:

   If target AWS Kinesis Data Streams does not exist, then it will be auto created by Oracle GoldenGate for Big Data. You can also use Template Keywords to dynamically assign kinesis data stream names.
6. If replicat starts successfully, then it will be in running state. You can go to action/details/statistics to see the replication statistics.

   Figure 7-46 If replicat starts successfully, it will be in running state. You can go to action/details/statistics to see the replication statistics.

   

   Figure 7-47 Replication Statistics

   
7. Go to AWS Kinesis console and check the data streams:

   Figure 7-48 In the AWS Kinesis console check the data streams.

   

Note:

• AWS Kinesis Event Handler can be configured for proxy server. For more information, see Configuring the Proxy Server for Kinesis Streams Handler.
• For more information about AWS Kinesis performance considerations, see Kinesis Handler Performance Considerations.
• For more information about AWS Kinesis input limitations, see Kinesis Streams Input Limits.