8.10 Realtime Message Ingestion to Google Pub/Sub with Oracle GoldenGate for Distributed Applications and Analytics

Overview

This Quickstart covers a step-by-step process showing how to ingest messages to Google Pub/Sub in real-time with Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA).

Google Pub/Sub is a scalable, reliable messaging service that allows asynchronous communication between applications by decoupling message producers (publishers) from message consumers (subscribers).

GG for DAA connects Google Pub/Sub with Google Pub/Sub Handler. GG for DAA reads the source operations from the trail file, formats them, maps to Google Pub/Sub topics and delivers.

• Prerequisites
  
• Install Dependency Files
  
• Create a Replicat in Oracle GoldenGate for Distributed Applications and Analytics
  

8.10.1 Prerequisites

• Google Pub/Sub service and topic
• Google Service Account Key

In this Quickstart, a sample trail file tr which is shipped with GG for DAA is used. The sample trail file is located at GG_HOME/opt/AdapterExamples/trail/ in your GG for DAA instance.

8.10.2 Install Dependency Files

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) uses Google Pub/Sub client libraries. You can use the Dependency Downloader to download the client libraries. Dependency Downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.

1. In your GG for DAA VM, go to Dependency Downloader utility. It is located at GG_HOME/opt/DependencyDownloader/. Locate googlepubsub.sh.
2. Execute googlepubsub.sh with the required version. You can check the version and reported vulnerabilities in Maven Central. This Quickstart uses 1.129.3, which is the latest version when it is published.
3. A new directory <googlepubsub_version> is created in GG_HOME/opt/DependencyDownloader/dependencies. Note this directory as it will be used in the replicat properties.
   • /u01/app/ogg/opt/DependencyDownloader/dependencies/googlepubsub_1.129.3

8.10.3 Create a Replicat in Oracle GoldenGate for Distributed Applications and Analytics

To create a replicat in Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA):

1. In the Administration Service tab, click the + sign to add a replicat.

   Figure 8-69 Click + in the Administration Service tab.

   
2. Select the Classic Replicat Replicat Type and click Next There are two different Replicat types available: Classic and Coordinated. Classic Replicat is a single threaded process whereas Coordinated Replicat is a multithreaded one that applies transactions in parallel.

   Figure 8-70 Add Replicat

   
3. Enter the Replicat options, and click Next:
   1. Replicat Trail: Name of the required trail file (if using sample trail, provide as tr)
   2. Subdirectory: Provide as GG_HOME/opt/AdapterExamples/trail/ if using the sample trail.
   3. Target: Google Pub/Sub

      Figure 8-71 Replicat Options

      
4. Leave Managed Options as is and click Next.

   Figure 8-72 Managed Options

   
5. Enter Parameter File details and click Next. In the Parameter File, you can specify source to target mapping or leave it as-is with a wildcard selection.

   Figure 8-73 Parameter File

   
6. In Properties File, update the properties marked as TODO and click Create and Run.

   # Properties file for Replicat GPUBSUB
   #Google Pub/Sub Handler Template
   gg.handlerlist=pubsub
   gg.handler.pubsub.type=googlepubsub
   gg.handler.pubsub.mode=op
   #TODO: Set the path to the JSON credenials file
   gg.handler.pubsub.credentialsFile=
   #TODO: Set the template to resolve the topic name
   gg.handler.pubsub.topicMappingTemplate=
   #TODO Set the project name
   gg.handler.pubsub.projectName=
   #TODO: Set the template to resolve the order key - Not required but recommended.
   gg.handler.pubsub.orderingKeyMappingTemplate= ${primaryKeys}
   gg.handler.pubsub.format=json_row
   gg.handler.pubsub.format.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]}
   #TODO: Set the path to the Google Pub/Sub client jar files.
   gg.classpath=path_to/dependencies/googlepubsub_1.129.3/*
   
   GG for DAA supports dynamic topic mapping by template keywords.  For example, if you assign topicMappingTemplate as  ${tablename}, GG for DAA will create a topic with the source table name, per each source table and will map the events to these topics.
   
   Oracle recommends using orderingkeyMappingTemplate=${primaryKeys}, GG for DAA will send the source operations with the same pk in the same source order. This will guarantee maintaining the order of the source operations while subscriber clients receive the messages.
   

   GG for DAA supports dynamic topic mapping by Template Keywords. For example, if you assign topicMappingTemplate as ${tablename}, GG for DAA will create a topic with the source table name, per each source table and will map the events to these topics.

   Oracle recommends using orderingkeyMappingTemplate=${primaryKeys}, GG for DAA sends the source operations with the same pk in the same source order. This guarantees maintaining the order of the source operations while subscriber clients receive the messages.

7. If replicat starts successfully, it will be in running state. You can go to action/details/statistics to see the replication statistics.

   Figure 8-74 Replication Statistics

   
8. You can go to your Google Pub/Sub and check the messages

For more information about Google Pub/Sub replication, see Google Pub/Sub.