8.10 Realtime Message Ingestion to Google Pub/Sub with Oracle GoldenGate for Distributed Applications and Analytics
Overview
This Quickstart covers a step-by-step process showing how to ingest messages to Google Pub/Sub in real-time with Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA).
Google Pub/Sub is a scalable, reliable messaging service that allows asynchronous communication between applications by decoupling message producers (publishers) from message consumers (subscribers).
GG for DAA connects Google Pub/Sub with Google Pub/Sub Handler. GG for DAA reads the source operations from the trail file, formats them, maps to Google Pub/Sub topics and delivers.
- Prerequisites
- Install Dependency Files
- Create a Replicat in Oracle GoldenGate for Distributed Applications and Analytics
Parent topic: Quickstarts
8.10.1 Prerequisites
- Google Pub/Sub service and topic
- Google Service Account Key
In this Quickstart, a sample trail file tr
which is shipped with GG
for DAA is used. The sample trail file is located at
GG_HOME/opt/AdapterExamples/trail/
in your GG for DAA
instance.
8.10.2 Install Dependency Files
Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) uses Google Pub/Sub client libraries. You can use the Dependency Downloader to download the client libraries. Dependency Downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.
- In your GG for DAA VM, go to Dependency Downloader utility. It is
located at
GG_HOME/opt/DependencyDownloader/
. Locategooglepubsub.sh
. - Execute
googlepubsub.sh
with the required version. You can check the version and reported vulnerabilities in Maven Central. This Quickstart uses 1.129.3, which is the latest version when it is published. - A new directory
<googlepubsub_version>
is created inGG_HOME/opt/DependencyDownloader/dependencies
. Note this directory as it will be used in the replicat properties./u01/app/ogg/opt/DependencyDownloader/dependencies/googlepubsub_1.129.3
8.10.3 Create a Replicat in Oracle GoldenGate for Distributed Applications and Analytics
To create a replicat in Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA):
- In the Administration Service tab, click the + sign
to add a replicat.
Figure 8-69 Click + in the Administration Service tab.
- Select the Classic Replicat Replicat Type and click
Next There are two different Replicat types available: Classic and
Coordinated. Classic Replicat is a single threaded process whereas Coordinated
Replicat is a multithreaded one that applies transactions in parallel.
Figure 8-70 Add Replicat
- Enter the Replicat options, and click Next:
- Replicat Trail: Name of the required trail file (if
using sample trail, provide as
tr
) - Subdirectory: Provide as
GG_HOME/opt/AdapterExamples/trail/
if using the sample trail. - Target: Google Pub/Sub
Figure 8-71 Replicat Options
- Replicat Trail: Name of the required trail file (if
using sample trail, provide as
- Leave Managed Options as is and click Next.
Figure 8-72 Managed Options
- Enter Parameter File details and click Next. In the
Parameter File, you can specify source to target mapping or leave it as-is with
a wildcard selection.
Figure 8-73 Parameter File
- In Properties File, update the properties marked as TODO and
click Create and
Run.
# Properties file for Replicat GPUBSUB #Google Pub/Sub Handler Template gg.handlerlist=pubsub gg.handler.pubsub.type=googlepubsub gg.handler.pubsub.mode=op #TODO: Set the path to the JSON credenials file gg.handler.pubsub.credentialsFile= #TODO: Set the template to resolve the topic name gg.handler.pubsub.topicMappingTemplate= #TODO Set the project name gg.handler.pubsub.projectName= #TODO: Set the template to resolve the order key - Not required but recommended. gg.handler.pubsub.orderingKeyMappingTemplate= ${primaryKeys} gg.handler.pubsub.format=json_row gg.handler.pubsub.format.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]} #TODO: Set the path to the Google Pub/Sub client jar files. gg.classpath=path_to/dependencies/googlepubsub_1.129.3/* GG for DAA supports dynamic topic mapping by template keywords. For example, if you assign topicMappingTemplate as ${tablename}, GG for DAA will create a topic with the source table name, per each source table and will map the events to these topics. Oracle recommends using orderingkeyMappingTemplate=${primaryKeys}, GG for DAA will send the source operations with the same pk in the same source order. This will guarantee maintaining the order of the source operations while subscriber clients receive the messages.
GG for DAA supports dynamic topic mapping by Template Keywords. For example, if you assigntopicMappingTemplate as ${tablename}
, GG for DAA will create a topic with the source table name, per each source table and will map the events to these topics.Oracle recommends using
orderingkeyMappingTemplate=${primaryKeys}
, GG for DAA sends the source operations with the same pk in the same source order. This guarantees maintaining the order of the source operations while subscriber clients receive the messages. - If replicat starts successfully, it will be in running state. You
can go to action/details/statistics to see the replication statistics.
Figure 8-74 Replication Statistics
- You can go to your Google Pub/Sub and check the messages
For more information about Google Pub/Sub replication, see Google Pub/Sub.