11.2.13 Apache Pulsar

• Overview
  
• Prerequisites
  
• Setting Up and Running the Pulsar Handler
  
• Pulsar Handler Configuration
  

11.2.13.1 Overview

The Apache Pulsar Handler sends formatted GoldenGate operation data to Pulsar topics. The target topic is selected by a GoldenGate mapping template, and the optional message key is selected by a key mapping template. Messages are sent asynchronously and flushed when Replicat commits a transaction.

11.2.13.2 Prerequisites

Before configuring the handler, make sure that:

• Oracle GoldenGate for Distributed Applications and Analytics is installed.
• The Replicat process can connect to the Pulsar broker service URL.
• The Pulsar topics used by Replicat already exist, or topic auto-creation is enabled in Pulsar.
• The Pulsar client libraries are available to Replicat in gg.classpath.
• If you run the Dependency Downloader utility, JAVA_HOME is set to a supported Java 17 JDK.

11.2.13.3 Setting Up and Running the Pulsar Handler

• Download the Pulsar Client Libraries
  
• Create the Pulsar Producer Configuration File
  
• Configure the Replicat Parameter File
  
• Configure the Pulsar Handler Properties
  

11.2.13.3.1 Download the Pulsar Client Libraries

Use the Dependency Downloader utility included with Oracle GoldenGate to download the Apache Pulsar client dependencies.

cd <OGG_HOME>/DependencyDownloader./pulsar.sh 4.0.8

The dependencies are downloaded to the following location:

<OGG_HOME>/DependencyDownloader/dependencies/pulsar_4.0.8/

Add that directory to the Replicat properties file with gg.classpath.

11.2.13.3.2 Create the Pulsar Producer Configuration File

Create a Pulsar producer configuration file that contains the Pulsar client properties. At minimum, configure the Pulsar broker service URL.

Example pulsar-client.properties:

serviceUrl=pulsar://pulsar-broker.example.com:6650

You can add other Pulsar client properties required by your environment, such as TLS or authentication properties. Do not configure GoldenGate handler properties in this file.

11.2.13.3.3 Configure the Replicat Parameter File

Create or edit the Replicat parameter file. The

TARGETDB LIBFILE

parameter specifies the Oracle GoldenGate Java Adapter properties file.

Example Replicat parameter file:

REPLICAT rpulsar
TARGETDB LIBFILE libggjava.so SET property=dirprm/pulsar.properties
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP source_schema.*, TARGET source_schema.*;

When using the Administration Service, specify the equivalent parameter values when creating or editing the Replicat.

11.2.13.3.4 Configure the Pulsar Handler Properties

Create the Oracle GoldenGate Java Adapter properties file referenced by the Replicat parameter file.

Example pulsar.properties:

gg.handlerlist=pulsar
gg.handler.pulsar.type=pulsar
gg.handler.pulsar.mode=op

gg.handler.pulsar.pulsarProducerConfigFile=/u01/ogg/dirprm/pulsar-client.properties

gg.handler.pulsar.topicMappingTemplate=persistent://public/default/${tableName}

gg.handler.pulsar.keyMappingTemplate=${primaryKeys}

gg.handler.pulsar.schemaTopicName=persistent://public/default/ogg-schema

gg.handler.pulsar.format=json
gg.handler.pulsar.format.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]}

gg.handler.pulsar.compressionType=lz4
gg.handler.pulsar.enableBatching=true
gg.handler.pulsar.sendTimeoutInSeconds=120

gg.classpath=/u01/ogg/DependencyDownloader/dependencies/pulsar_4.0.8/*

Use any supported Oracle GoldenGate formatter that matches your downstream consumer requirements. For example, use the JSON formatter for applications that consume JSON payloads, or an Avro formatter when the downstream application expects Avro-formatted messages.

11.2.13.4 Pulsar Handler Configuration

Pulsar Handler Configuration describes the properties required to configure the Pulsar Handler for data replication to Apache Pulsar.

• Pulsar Handler Properties
  

11.2.13.4.1 Pulsar Handler Properties

The Pulsar Handler properties use the following naming convention:

gg.handler.<handler-name>.<property-name>

For the examples in this topic, <handler-name> is pulsar.

Table 11-13 Pulsar Handler Properties Configuration

Properties	Required	Default	Description
gg.handlerlist	Required	None	Lists the configured handler names.
gg.handler.<handler-name>.type	Required	None	Specifies the handler type. Set the value to pulsar.
gg.handler.<handler-name>.mode	Optional	op	Specifies the handler mode. Use op to publish each operation as an individual Pulsar message.
gg.handler.<handler-name>.pulsarProducerConfigFile	Required	None	Specifies the path to the Pulsar producer configuration file.
gg.handler.<handler-name>.topicMappingTemplate	Required	None	Specifies the GoldenGate template used to resolve the target Pulsar topic.
gg.handler.<handler-name>.keyMappingTemplate	Optional	${null}	Specifies the GoldenGate template used to resolve the Pulsar message key
gg.handler.<handler-name>.schemaTopicName	Optional	None	Specifies the Pulsar topic used to publish formatter schema messages.
gg.handler.<handler-name>.metaHeadersTemplate	Optional	None	Specifies the GoldenGate metadata template used to populate Pulsar message properties.
gg.handler.<handler-name>.compressionType	Optional	None	Specifies the compression type. Valid values are none, lz4, zlib, zstd, and snappy.
gg.handler.<handler-name>.accessMode	Optional	Shared	Specifies the producer access mode. Valid values are Shared, Exclusive, and WaitForExclusive.
gg.handler.<handler-name>.autoUpdatePartitions	Optional	true	Enables automatic discovery of new partitions.
gg.handler.<handler-name>.autoUpdatePartitionsIntervalSeconds	Optional	60	Specifies the interval, in seconds, between partition discovery operations.
gg.handler.<handler-name>.hashingScheme	Optional	JavaStringHash	Specifies the hashing scheme used for partitioned topics.
gg.handler.<handler-name>.enableBatching	Optional	true	Enables Pulsar producer batching.
gg.handler.<handler-name>.enableChunking	Optional	false	Enables message chunking for large messages. Disable batching when chunking is enabled.
gg.handler.<handler-name>.enableLazyStartPartitionedProducers	Optional	true	Enables lazy initialization of partitioned producers.
gg.handler.<handler-name>.batchingMaxBytes	Optional	Pulsar client default	Specifies the maximum batch size, in bytes.
gg.handler.<handler-name>.batchingMaxMessages	Optional	Pulsar client default	Specifies the maximum number of messages in a batch.
gg.handler.<handler-name>.maxPendingMessages	Optional	0	Specifies the maximum number of pending messages. A value of 0 indicates no handler-configured limit.
gg.handler.<handler-name>.sendTimeoutInSeconds	Optional	120	Specifies the time, in seconds, to wait for a broker acknowledgment before reporting an error.
gg.handler.<handler-name>.blockIfQueueFull	Optional	true	Specifies whether the producer blocks when the pending message queue is full.
gg.classpath	Required	None	Specifies the location of the Pulsar client libraries and any additional required third-party libraries.