9.2.22 Google Pub/Sub
- Overview
- Detailed Functionality
- Setting up and Running the Google PubSub Handler
- Configuring Handler Authentication
- Google PubSub Handler Configuration
- Proxy Settings
- Sample Configuration
- Google PubSub Dependencies
Parent topic: Target
9.2.22.1 Overview
Pub/Sub allows services to communicate asynchronously, with latencies on the order of 100 milliseconds.
Pub/Sub enables you to create systems of event producers and consumers, called publishers and subscribers. Publishers communicate with subscribers asynchronously by broadcasting events, rather than by synchronous remote procedure calls (RPCs).
Publishers send events to the Pub/Sub service, without regard to how or when these events are to be processed. Pub/Sub then delivers events to all the services that react to them. In systems communicating through RPCs, publishers must wait for subscribers to receive the data. However, the asynchronous integration in Pub/Sub increases the flexibility and robustness of the overall system.
Parent topic: Google Pub/Sub
9.2.22.2 Detailed Functionality
Transaction Mode
The following configuration sets the Google PubSub Handler to transaction mode:gg.handler.name.Mode=tx
. In transaction
mode, the serialized data is concatenated for every operation in a transaction from the source
Oracle GoldenGate trail files. The contents of the concatenated operation data is the value of
the Google Pubsub message. The result is that the messages comprise data from 1 to N
operations, where N is the number of operations in the transaction.
Operation Mode
The following configuration sets the GooglePubSub Handler to operation mode:gg.handler.name.Mode=op
.
In operation mode, the serialized data for each operation is placed into an individual Google Pubsub message as the value. This means that there is a 1 to 1 relationship between the incoming operations and the number of Google Pubsub messages produced.
Parent topic: Google Pub/Sub
9.2.22.3 Setting up and Running the Google PubSub Handler
Configuring Classpath
For the Google PubSub Handler to connect to Google PubSub and run, the properties file and the Google PubSub client JARs must be configured in thegg.classpath
configuration variable. For the
GooglePubSub
jar files use the dependency downloader tool.
The
recommended storage location for the Google PubSub properties file is the Oracle GoldenGate
dirprm
directory. The gg.classpath
must be configured
precisely.
*
wildcard is included in the path to the
properties file, then the file is not picked up. Conversely, path to the dependency JARs must
include the *
wild card character in order to include all the JAR files in
that directory in the associated classpath. Do not use *.jar
. The following
is an example of the correctly configured classpath: gg.classpath={google PubSub
install dir}/libs/*
.
Parent topic: Google Pub/Sub
9.2.22.4 Configuring Handler Authentication
You have to configure the Google PubSub Handler authentication using the credentials in the JSON file downloaded from Google PubSub.
Download the credentials file:
- Login into your Google account at cloud.google.com.
- Click Console, and then to go to the Dashboard where you can select your project.
- From the navigation menu, click APIs & Services, and then select Credentials.
- From the Create Credentials menu, choose Service account key.
- Choose the JSON key type to download the JSON credentials file for your system.
Specify the path to the credentials file in the properties file with the gg.handler.name.credentialsFile configuration property.
The path of the credentials file must contain the path with no wildcard appended. If you include the*
wildcard in the path to the credentials file, then the file is
not recognized.
Google PubSub credentials file configuration sample:
{ "type": "<accountType>", "project_id": "<projectType>", "private_key_id": "<privateKeyId>", "private_key": "<privateKey>", "client_email": "<googleCloudSubscriptionEmailId>", "client_id": "<subscriptionClientId>", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "", "client_x509_cert_url": "" }
Parent topic: Google Pub/Sub
9.2.22.5 Google PubSub Handler Configuration
Property Name | Required/Optional | Property Value | Default | Description |
gg.handlerlist |
Required | name (choice of any name) | None | List of handlers to be used |
gg.handler.name.type |
Required | oracle.goldengate.handler.googlepubsub.GooglePubsubHandler |
None | Type of handler to use |
gg.handler.name.format |
Optional | Formatter class or short code | json | Formatter to use to format payload. Can be one of
xml , delimitedtext ,
json , json_row ,
avro_row , or avro_op |
gg.handler.name.credentialsFile |
Required | The name of the credentials file in json format with absolute path | json | NA |
gg.handler.name mode |
Optional | tx/op |
tx | NA |
gg.handler.name.topicMappingTemplate |
Required | NA | NA | Google PubSub Topic name to which the replicat will publish messages to. See Using Templates to Resolve the Topic Name and Message Key |
gg.handler.name.projectName |
Required | NA | NA | Google PubSub project name |
gg.handler.name.batchProcessing |
Optional | true /false |
false |
When enabled the messages will be processed in batches enhancing the performance replicat. |
gg.handler.name.requestBytesThreshold |
Optional | Numeric integer value | 1000 | Maximum size of publish request by specifying the maximum number of bytes |
gg.handler.name.messageCountBatchSize |
Optional | Numeric integer value | 100 | Maximum size of publish request by specifying the maximum number of messages |
gg.handler.name.publishDelayThreshold |
Optional | Numeric integer value | 100 | Provides flexibility to control how long to wait before sending a batch, specifically in regard to the amount of time messages are held in order to fill batches. Decreasing this value improves latency. This property value is the batch-time interval in milliseconds. |
gg.handler.name.attributesTemplate |
Optional | Comma delimited list of attribute keywords. | None | The different properties of the message can be set as attributes to the google pubsub message. This facilitates the user/subscriber to filter messages based on the set attributes. See Metacolumn Keywords for more information about configuring this property. |
gg.handler.
name.orderingKeyMappingTemplate |
Optional |
A template string value to resolve the pubsub message ordering key at runtime. When running transaction mode use a static
|
None | See Using Templates to Resolve the Topic Name and Message Key |
Parent topic: Google Pub/Sub
9.2.22.6 Proxy Settings
jvm.bootoptions= -Dhttps.proxyHost=proxy_host_name -Dhttps.proxyPort=proxy_port_number
Parent topic: Google Pub/Sub
9.2.22.7 Sample Configuration
gg.handlerlist=pubsub #The Google Pub/Sub Handler gg.handler.pubsub.type=googlepubsub gg.handler.pubsub.mode=op #Set the path to the JSON credenials file gg.handler.pubsub.credentialsFile= #Set the template to resolve the topic name gg.handler.pubsub.topicMappingTemplate= #Set the project name gg.handler.pubsub.projectName= #Set the template to resolve the order key gg.handler.pubsub.orderingKeyMappingTemplate= gg.handler.pubsub.format=json_row gg.handler.pubsub.format.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]}
Parent topic: Google Pub/Sub
9.2.22.8 Google PubSub Dependencies
The Google PubSub client libraries are required for integration with Google PubSub. The maven coordinates are as follows:
Maven groupId: com.google.cloud
Maven artifactId: google-cloud-pubsub
api-common-2.31.0.jar guava-33.1.0-jre.jar protobuf-java-util-3.25.3.jar listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar proto-google-iam-v1-1.34.0.jar threetenbp-1.6.9.jar google-cloud-pubsub-1.129.3.jar opencensus-proto-0.2.0.jar google-http-client-gson-1.44.1.jar grpc-protobuf-lite-1.62.2.jar grpc-util-1.62.2.jar javax.annotation-api-1.3.2.jar checker-qual-3.42.0.jar error_prone_annotations-2.26.1.jar gax-grpc-2.48.0.jar httpcore-4.4.16.jar proto-google-common-protos-2.39.0.jar google-auth-library-oauth2-http-1.23.0.jar opencensus-contrib-http-util-0.31.1.jar protobuf-java-3.25.3.jar proto-google-cloud-pubsub-v1-1.111.3.jar grpc-api-1.62.2.jar perfmark-api-0.27.0.jar gax-2.48.0.jar jsr305-3.0.2.jar conscrypt-openjdk-uber-2.5.2.jar failureaccess-1.0.2.jar grpc-inprocess-1.62.2.jar grpc-grpclb-1.62.2.jar grpc-netty-shaded-1.62.2.jar google-http-client-1.44.1.jar grpc-core-1.62.2.jar j2objc-annotations-3.0.0.jar commons-codec-1.16.1.jar gson-2.10.1.jar grpc-auth-1.62.2.jar google-auth-library-credentials-1.23.0.jar grpc-services-1.62.2.jar grpc-context-1.62.2.jar animal-sniffer-annotations-1.23.jar opencensus-api-0.31.1.jar gax-httpjson-2.48.0.jar auto-value-annotations-1.10.4.jar grpc-stub-1.62.2.jar annotations-4.1.1.4.jar grpc-xds-1.62.2.jar grpc-alts-1.62.2.jar grpc-googleapis-1.62.2.jar httpclient-4.5.14.jar re2j-1.7.jar grpc-protobuf-1.62.2.jar
Parent topic: Google Pub/Sub