1. Oracle GoldenGate for Distributed Applications and Analytics
  2. Replicate Data
  3. Target
  4. Google Pub/Sub

9.2.22 Google Pub/Sub

Parent topic: Target

9.2.22.1 Overview

Pub/Sub allows services to communicate asynchronously, with latencies on the order of 100 milliseconds.

Pub/Sub enables you to create systems of event producers and consumers, called publishers and subscribers. Publishers communicate with subscribers asynchronously by broadcasting events, rather than by synchronous remote procedure calls (RPCs).

Publishers send events to the Pub/Sub service, without regard to how or when these events are to be processed. Pub/Sub then delivers events to all the services that react to them. In systems communicating through RPCs, publishers must wait for subscribers to receive the data. However, the asynchronous integration in Pub/Sub increases the flexibility and robustness of the overall system.

Parent topic: Google Pub/Sub

9.2.22.2 Detailed Functionality

Transaction Mode

The following configuration sets the Google PubSub Handler to transaction mode: gg.handler.name.Mode=tx. In transaction mode, the serialized data is concatenated for every operation in a transaction from the source Oracle GoldenGate trail files. The contents of the concatenated operation data is the value of the Google Pubsub message. The result is that the messages comprise data from 1 to N operations, where N is the number of operations in the transaction.

Operation Mode

The following configuration sets the GooglePubSub Handler to operation mode: gg.handler.name.Mode=op.

In operation mode, the serialized data for each operation is placed into an individual Google Pubsub message as the value. This means that there is a 1 to 1 relationship between the incoming operations and the number of Google Pubsub messages produced.

Parent topic: Google Pub/Sub

9.2.22.3 Setting up and Running the Google PubSub Handler

Configuring Classpath

For the Google PubSub Handler to connect to Google PubSub and run, the properties file and the Google PubSub client JARs must be configured in the gg.classpath configuration variable. For the GooglePubSub jar files use the dependency downloader tool.

The recommended storage location for the Google PubSub properties file is the Oracle GoldenGate dirprm directory. The gg.classpath must be configured precisely.

The path of the Google PubSub properties file must contain the path with no wildcard appended. If the * wildcard is included in the path to the properties file, then the file is not picked up. Conversely, path to the dependency JARs must include the * wild card character in order to include all the JAR files in that directory in the associated classpath. Do not use *.jar. The following is an example of the correctly configured classpath: gg.classpath={google PubSub install dir}/libs/*.

Parent topic: Google Pub/Sub

9.2.22.4 Configuring Handler Authentication

You have to configure the Google PubSub Handler authentication using the credentials in the JSON file downloaded from Google PubSub.

Download the credentials file:

  1. Login into your Google account at cloud.google.com.
  2. Click Console, and then to go to the Dashboard where you can select your project.
  3. From the navigation menu, click APIs & Services, and then select Credentials.
  4. From the Create Credentials menu, choose Service account key.
  5. Choose the JSON key type to download the JSON credentials file for your system.

Specify the path to the credentials file in the properties file with the gg.handler.name.credentialsFile configuration property.

The path of the credentials file must contain the path with no wildcard appended. If you include the * wildcard in the path to the credentials file, then the file is not recognized.

Google PubSub credentials file configuration sample: 

{
  "type": "<accountType>",
  "project_id": "<projectType>",
  "private_key_id": "<privateKeyId>",
  "private_key": "<privateKey>",
  "client_email": "<googleCloudSubscriptionEmailId>",
  "client_id": "<subscriptionClientId>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "",
  "client_x509_cert_url": ""
}

Parent topic: Google Pub/Sub

9.2.22.5 Google PubSub Handler Configuration

Property Name Required/Optional Property Value Default Description
gg.handlerlist Required name (choice of any name) None List of handlers to be used
gg.handler.name.type Required oracle.goldengate.handler.googlepubsub.GooglePubsubHandler None Type of handler to use
gg.handler.name.format Optional Formatter class or short code json Formatter to use to format payload. Can be one of xml, delimitedtext, json, json_row, avro_row, or avro_op
gg.handler.name.credentialsFile Required The name of the credentials file in json format with absolute path json NA
gg.handler.name mode Optional tx/op tx NA
gg.handler.name.topicMappingTemplate Required NA NA Google PubSub Topic name to which the replicat will publish messages to. See Using Templates to Resolve the Topic Name and Message Key
gg.handler.name.projectName Required NA NA Google PubSub project name
gg.handler.name.batchProcessing Optional true/false false When enabled the messages will be processed in batches enhancing the performance replicat.
gg.handler.name.requestBytesThreshold Optional Numeric integer value 1000 Maximum size of publish request by specifying the maximum number of bytes
gg.handler.name.messageCountBatchSize Optional Numeric integer value 100 Maximum size of publish request by specifying the maximum number of messages
gg.handler.name.publishDelayThreshold Optional Numeric integer value 100 Provides flexibility to control how long to wait before sending a batch, specifically in regard to the amount of time messages are held in order to fill batches. Decreasing this value improves latency. This property value is the batch-time interval in milliseconds.
gg.handler.name.attributesTemplate Optional Comma delimited list of attribute keywords. None The different properties of the message can be set as attributes to the google pubsub message. This facilitates the user/subscriber to filter messages based on the set attributes. See Metacolumn Keywords for more information about configuring this property.
gg.handler. name.orderingKeyMappingTemplate Optional

A template string value to resolve the pubsub message ordering key at runtime.

When running transaction mode use a static orderingKey to order the messages.

None See Using Templates to Resolve the Topic Name and Message Key

Parent topic: Google Pub/Sub

9.2.22.6 Proxy Settings

To connect to Google PubSub using a proxy server, you must configure the proxy host and the proxy port in the properties file as follows:
jvm.bootoptions= -Dhttps.proxyHost=proxy_host_name -Dhttps.proxyPort=proxy_port_number

Parent topic: Google Pub/Sub

9.2.22.7 Sample Configuration

The following is a sample configuration for the Google PubSub Handler:
gg.handlerlist=pubsub
#The Google Pub/Sub Handler
gg.handler.pubsub.type=googlepubsub
gg.handler.pubsub.mode=op
#Set the path to the JSON credenials file
gg.handler.pubsub.credentialsFile=
#Set the template to resolve the topic name
gg.handler.pubsub.topicMappingTemplate=
#Set the project name
gg.handler.pubsub.projectName=
#Set the template to resolve the order key
gg.handler.pubsub.orderingKeyMappingTemplate=
gg.handler.pubsub.format=json_row
gg.handler.pubsub.format.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]}

Parent topic: Google Pub/Sub

9.2.22.8 Google PubSub Dependencies

The Google PubSub client libraries are required for integration with Google PubSub. The maven coordinates are as follows:

Maven groupId: com.google.cloud

Maven artifactId: google-cloud-pubsub

Version: 1.129.3
api-common-2.31.0.jar
guava-33.1.0-jre.jar
protobuf-java-util-3.25.3.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
proto-google-iam-v1-1.34.0.jar
threetenbp-1.6.9.jar
google-cloud-pubsub-1.129.3.jar
opencensus-proto-0.2.0.jar
google-http-client-gson-1.44.1.jar
grpc-protobuf-lite-1.62.2.jar
grpc-util-1.62.2.jar
javax.annotation-api-1.3.2.jar
checker-qual-3.42.0.jar
error_prone_annotations-2.26.1.jar
gax-grpc-2.48.0.jar
httpcore-4.4.16.jar
proto-google-common-protos-2.39.0.jar
google-auth-library-oauth2-http-1.23.0.jar
opencensus-contrib-http-util-0.31.1.jar
protobuf-java-3.25.3.jar
proto-google-cloud-pubsub-v1-1.111.3.jar
grpc-api-1.62.2.jar
perfmark-api-0.27.0.jar
gax-2.48.0.jar
jsr305-3.0.2.jar
conscrypt-openjdk-uber-2.5.2.jar
failureaccess-1.0.2.jar
grpc-inprocess-1.62.2.jar
grpc-grpclb-1.62.2.jar
grpc-netty-shaded-1.62.2.jar
google-http-client-1.44.1.jar
grpc-core-1.62.2.jar
j2objc-annotations-3.0.0.jar
commons-codec-1.16.1.jar
gson-2.10.1.jar
grpc-auth-1.62.2.jar
google-auth-library-credentials-1.23.0.jar
grpc-services-1.62.2.jar
grpc-context-1.62.2.jar
animal-sniffer-annotations-1.23.jar
opencensus-api-0.31.1.jar
gax-httpjson-2.48.0.jar
auto-value-annotations-1.10.4.jar
grpc-stub-1.62.2.jar
annotations-4.1.1.4.jar
grpc-xds-1.62.2.jar
grpc-alts-1.62.2.jar
grpc-googleapis-1.62.2.jar
httpclient-4.5.14.jar
re2j-1.7.jar
grpc-protobuf-1.62.2.jar

Parent topic: Google Pub/Sub