Configure

Configuring Oracle GoldenGate for Distributed Applications and Analytics

This topic describes how to configure GG for DAA handlers.

Running with Replicat

You need to review before configuring a replicat process in Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA).

This topic explains how to run the Java Adapter with the Oracle GoldenGate Replicat process.

Replicat Grouping

The Replicat process provides the Replicat configuration property, GROUPTRANSOPS, to control transaction grouping. By default, the Replicat process implements transaction grouping of 1000 source transactions into a single target transaction. If you want to turn off transaction grouping then the GROUPTRANSOPS Replicat property should be set to 1.

About Replicat Checkpointing

In addition to the Replicat checkpoint file ,.cpr, an additional checkpoint file, dirchk/group.cpj, is created that contains information similar to CHECKPOINTTABLE in Replicat for the database.

About Initial Load Support

Replicat can already read trail files that come from both the online capture and initial load processes that write to a set of trail files. In addition, Replicat can also be configured to support the delivery of the special run initial load process using RMTTASK specification in the Extract parameter file. For more details about configuring the direct load, see Initial Load Extract.

Note: The SOURCEDB or DBLOGIN parameter specifications vary depending on your source database.

About the Unsupported Replicat Features

The following Replicat features are not supported in this release:

How the Mapping Functionality Works

The Oracle GoldenGate Replicat process supports mapping functionality to custom target schemas. You must use the Metadata Provider functionality to define a target schema or schemas, and then use the standard Replicat mapping syntax in the Replicat configuration file to define the mapping.

About Schema Evolution and Metadata Change Events

The Metadata in trail is a feature that allows seamless runtime handling of metadata change events by Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA), including schema evolution and schema propagation to GG for DAA target applications. The NO_OBJECTDEFS is a sub-parameter of the Extract and Replicat EXTTRAIL and RMTTRAIL parameters that lets you suppress the important metadata in trail feature and revert to using a static metadata definition.

The Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) Handlers and Formatters provide functionality to take action when a metadata change event is encountered. The ability to take action in the case of metadata change events depends on the metadata change events being available in the source trail file. Oracle GoldenGate supports metadata in trail and the propagation of DDL data from a source Oracle Database. If the source trail file does not have metadata in trail and DDL data (metadata change events) then it is not possible for GG for DAA to provide and metadata change event handling.

About Configuration Property CDATA[] Wrapping

The Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) Handlers and Formatters support the configuration of many parameters in the Java properties file, the value of which may be interpreted as white space. The configuration handling of the Java Adapter trims white space from configuration values from the Java configuration file. This behavior of trimming whitespace may be desirable for some configuration values and undesirable for other configuration values. Alternatively, you can wrap white space values inside of special syntax to preserve the whites pace for selected configuration variables. GG for DAA borrows the XML syntax of CDATA[] to preserve white space. Values that would be considered to be white space can be wrapped inside of CDATA[].

The following is an example attempting to set a new-line delimiter for the Delimited Text Formatter:

gg.handler.{name}.format.lineDelimiter=\n

This configuration will not be successful. The new-line character is interpreted as white space and will be trimmed from the configuration value. Therefore the gg.handler setting effectively results in the line delimiter being set to an empty string.

In order to preserve the configuration of the new-line character simply wrap the character in the CDATA[] wrapper as follows:

gg.handler.{name}.format.lineDelimiter=CDATA[\n]

Configuring the property with the CDATA[] wrapping preserves the white space and the line delimiter will then be a new-line character.

Using Regular Expression Search and Replace

You can perform more powerful search and replace operations of both schema data (catalog names, schema names, table names, and column names) and column value data, which are separately configured. Regular expressions (regex) are characters that customize a search string through pattern matching.

You can match a string against a pattern or extract parts of the match. Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) uses the standard Oracle Java regular expressions package, java.util.regex, see Regular Expressions in The Single UNIX Specification, Version 4.

Using Schema Data Replace

You can replace schema data using the gg.schemareplaceregex and gg.schemareplacestring properties. Use gg.schemareplaceregex to set a regular expression, and then use it to search catalog names, schema names, table names, and column names for corresponding matches. Matches are then replaced with the content of the gg.schemareplacestring value. The default value of gg.schemareplacestring is an empty string or "".

For example, some system table names start with a dollar sign like $mytable. You may want to replicate these tables even though most technologies do not allow dollar signs in table names. To remove the dollar sign, you could configure the following replace strings:

gg.schemareplaceregex=[$]
gg.schemareplacestring=

The resulting example of searched and replaced table name is mytable. These properties also support CDATA[] wrapping to preserve whitespace in the value of configuration values. So the equivalent of the preceding example using CDATA[] wrapping use is:

gg.schemareplaceregex=CDATA[[$]]
gg.schemareplacestring=CDATA[]

The schema search and replace functionality supports using multiple search regular expressions and replacements strings using the following configuration syntax:

gg.schemareplaceregex=some_regex
gg.schemareplacestring=some_value
gg.schemareplaceregex1=some_regex
gg.schemareplacestring1=some_value
gg.schemareplaceregex2=some_regex
gg.schemareplacestring2=some_value

Using Content Data Replace

You can replace content data using the gg.contentreplaceregex and gg.contentreplacestring properties to search the column values using the configured regular expression and replace matches with the replacement string. For example, this is useful to replace line feed characters in column values. If the delimited text formatter is used then line feeds occurring in the data will be incorrectly interpreted as line delimiters by analytic tools.

You can configure n number of content replacement regex search values. The regex search and replacements are done in the order of configuration. Configured values must follow a given order as follows:

gg.contentreplaceregex=some_regex
gg.contentreplacestring=some_value
gg.contentreplaceregex1=some_regex
gg.contentreplacestring1=some_value
gg.contentreplaceregex2=some_regex
gg.contentreplacestring2=some_value

Configuring a subscript of 3 without a subscript of 2 would cause the subscript 3 configuration to be ignored.

Div:

Attention:

Regular express searches and replacements require computer processing and can reduce the performance of the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) process.

To replace line feeds with a blank character you could use the following property configurations:

gg.contentreplaceregex=[\n]
gg.contentreplacestring=CDATA[ ]

This changes the column value from:

this is
me

to :

this is me

Both values support CDATA wrapping. The second value must be wrapped in a CDATA[] wrapper because a single blank space will be interpreted as whitespace and trimmed by the GG for DAA configuration layer. In addition, you can configure multiple search a replace strings. For example, you may also want to trim leading and trailing white space out of column values in addition to trimming line feeds from:

^\s+|\s+$
gg.contentreplaceregex1=^\s+|\s+$
gg.contentreplacestring1=CDATA[]

Scaling Oracle GoldenGate for Distributed Applications and Analytics Delivery

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) supports breaking down the source trail files into either multiple Replicat processes or by using Coordinated Delivery to instantiate multiple Java Adapter instances inside a single Replicat process to improve throughput. This allows you to scale GG for DAA delivery.

There are some cases where the throughput to GG for DAA integration targets is not sufficient to meet your service level agreements even after you have tuned your handler for maximum performance. When this occurs, you can configure parallel processing and delivery to your targets using one of the following methods:

With either method, you can split the data into parallel processing for improved throughput. Oracle recommends breaking the data down in one of the following two ways:

Additional limitations:

Example 7-1 Scaling Support for the Oracle GoldenGate for Distributed Applications and Analytics Handlers

Handler Name Splitting Source Data By Source Table Splitting Source Table Data into Sub Streams
Cassandra Supported

Supported when:

  • Required target tables in Cassandra are pre-created.

  • Metadata change events do not occur.

Elastic Search Supported Supported
HBase Supported when all required HBase namespaces are pre-created in HBase.

Supported when:

  • All required HBase namespaces are pre-created in HBase.

  • All required HBase target tables are pre-created in HBase. Schema evolution is not an issue because HBase tables have no schema definitions so a source metadata change does not require any schema change in HBase.

  • The source data does not contain any truncate operations.

HDFS Supported

Supported with some restrictions.

  • You must select a naming convention for generated HDFS files wherethe file names do not collide. Colliding HDFS file names results in a Replicat abend. When using coordinated apply it is suggested that you configure ${groupName} as part of the configuration for the gg.handler.name.fileNameMappingTemplate property . The ${groupName} template resolves to the Replicat name concatenated with the Replicat thread number, which provides unique naming per Replicat thread. For more information, see Coordinated Apply Support.

  • Schema propagatation to HDFS and Hive integration is not currently supported.

JDBC Supported Supported
Kafka Supported Supported for formats that support schema propagation, such as Avro. This is less desirable due to multiple instances feeding the same schema information to the target.
Kafka Connect Supported Supported
Kinesis Streams Supported Supported
MongoDB Supported Supported
Java File Writer Supported Supported with the following restrictions: <p>You must select a naming convention for generated files where the file names do not collide. Colliding file names may results in a Replicat abend and/or polluted data. When using coordinated apply it is suggested that you configure ${groupName} as part of the configuration for the gg.handler.name.fileNameMappingTemplate property . The ${groupName} template resolves to the Replicat name concatenated with the Replicat thread number, which provides unique naming per Replicat thread. For more information, see Coordinated Apply Support. </p>

Coordinated Apply Support

Support Matrix for Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) Targets

Feature index

Target Option1 Option2 Option3 Option4
Snowflake stage/merge [✓] [✓] [✓] *SNOW[✓]
BigQuery stage/merge [✓] [✓] [✓] *BQ [✓]
Databricks [✓] *DBX [✓] [✓] *DBX [✓]
ADW [✓] [✓] [✓] [✓]
Synapse [✓] [✓] [✓] *SYN[✓]
Redshift [✓] *RS[✓] [✓] *RS[✓]
OneLake [✓] [x] [✓] [x]
Iceberg [✓] *ICE[✓] [✓] *ICE[✓]
Snowflake streaming [✓] *TC [✓] [✓] *TC [✓]
BigQuery streaming [✓] *TC [✓] [✓] *TC[✓]
Cassandra [✓] *TC [✓] [✓] *TC [✓]
HBase [✓] *HBASE [✓] [✓] *HBASE [✓]
Elasticsearch [✓] [✓] [✓] [✓]
Kafka [✓] [✓] [✓] [✓]
Oracle Streaming [✓] [✓] [✓] [✓]
Kafka Connect [✓] [✓] [✓] [✓]
MongoDB [✓] [✓] [✓] [✓]
AJD [✓] [✓] [✓] [✓]
CosmosDB [✓] [✓] [✓] [✓]
Kinesis [✓] [✓] [✓] [✓]
Kafka Rest Proxy [✓] [✓] [✓] [✓]
Oracle NoSQL [✓] *TC [✓] [✓] *TC [✓]
Redis [✓] *REDIS [✓] [✓] *REDIS [✓]
Google Pub/Sub [✓] [✓] [✓] [✓]
File Writer [✓] *FW [✓] [✓] *FW [✓]
Hadoop (HDFS) [✓] *FW [✓] [✓] *FW [✓]
JDBC [✓] [✓] [✓] [✓]

*SNOW

*BQ

*DBX

*SYN

*RS

*ICE

*TC

*HBASE

*REDIS

*FW

Configuring Cluster High Availability

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) doesn’t have built-in high availability functionality. You need to use a standard cluster software’s high availability capability to provide the high availability functionality.

You can configure a high availability scenario on a cluster so that if the leader instance of (GG for DAA) on machine fails, another GG for DAA instance could be started on another machine to resume where the failed instance left off.

If you manually configure your instances to share common GG for DAA and Oracle GoldenGate files using a shared disk architecture you can create a fail over situation. For a cluster installation, these files would need to accessible from all machines and accessible in the same location.

The configuration files that must be shared are:

Checkpoint files must be shared for the ability to resume processing:

Using Identities in Oracle GoldenGate Credential Store

The Oracle GoldenGate credential store manages user IDs and their encrypted passwords (together known as credentials) that are used by Oracle GoldenGate processes to interact with the local database. The credential store eliminates the need to specify user names and clear-text passwords in the Oracle GoldenGate parameter files.

An optional alias can be used in the parameter file instead of the user ID to map to a userid and password pair in the credential store. The credential store is implemented as an auto login wallet within the Oracle Credential Store Framework (CSF). The use of an LDAP directory is not supported for the Oracle GoldenGate credential store. The auto login wallet supports automated restarts of Oracle GoldenGate processes without requiring human intervention to supply the necessary passwords.

In Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA), you specify the alias and domain in the property file not the actual user ID or password. User credentials are maintained in secure wallet storage.

Creating a Credential Store

You can create a credential store for your Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) environment.

Run the GGSCI ADD CREDENTIALSTORE command to create a file called cwallet.sso in the dircrd/ subdirectory of your Oracle GoldenGate installation directory (the default).

You can the location of the credential store (cwallet.sso file by specifying the desired location with the CREDENTIALSTORELOCATION parameter in the GLOBALS file. Note: Only one credential store can be used for each Oracle GoldenGate instance.

Adding Users to a Credential Store

After you create a credential store for your Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) environment, you can added users to the store.

Run the GGSCI ALTER CREDENTIALSTORE ADD USER userid PASSWORD password [ALIAS alias] [DOMAIN domain] command to create each user, where:

For example:

ALTER CREDENTIALSTORE ADD USER scott PASSWORD tiger ALIAS scsm2 domain ggadapters

Configuring Properties to Access the Credential Store

The Oracle GoldenGate Java Adapter properties file requires specific syntax to resolve user name and password entries in the Credential Store at runtime. For resolving a user name the syntax is the following:

ORACLEWALLETUSERNAME[alias domain_name]

For resolving a password the syntax required is the following:

ORACLEWALLETPASSWORD[alias domain_name]

The following example illustrate how to configure a Credential Store entry with an alias of myalias and a domain of mydomain.

Note: With HDFS Hive JDBC the user name and password is encrypted.

Oracle Wallet integration only works for configuration properties which contain the string username or password. For example:

gg.handler.hdfs.hiveJdbcUsername=ORACLEWALLETUSERNAME[myalias mydomain]
gg.handler.hdfs.hiveJdbcPassword=ORACLEWALLETPASSWORD[myalias mydomain]

ORACLEWALLETUSERNAME and ORACLEWALLETPASSWORD can be used in the Extract (similar to Replicat) in JMS handler as well. For example:

gg.handler.<name>.user=ORACLEWALLETUSERNAME[JMS_USR JMS_PWD]

gg.handler.<name>.password=ORACLEWALLETPASSWORD[JMS_USR JMS_PWD]

Consider the user name and password entries as accessible values in the Credential Store. Any configuration property resolved in the Java Adapter layer (not accessed in the C user exit layer) can be resolved from the Credential Store. This allows you more flexibility to be creative in how you protect sensitive configuration entries.

Logging

Logging is essential to troubleshooting Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) integrations with GG for DAA targets.

This topic details how GG for DAA integration log and the best practices for logging.

About Replicat Process Logging

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) integrations leverage the Java Delivery functionality. In this setup, either a Oracle GoldenGate Replicat process loads a user exit shared library. This shared library then loads a Java virtual machine to thereby interface with targets providing a Java interface. So the flow of data is as follows:

Replicat Process ->User Exit-> Java Layer

It is important that all layers log correctly so that users can review the logs to troubleshoot new installations and integrations. Additionally, if you have a problem that requires contacting Oracle Support, the log files are a key piece of information to be provided to Oracle Support so that the problem can be efficiently resolved.

A running Replicat process creates or appends log files into the GoldenGate_Home/dirrpt directory that adheres to the following naming convention: process_name.rpt. If a problem is encountered when deploying a new Oracle GoldenGate process, this is likely the first log file to examine for problems. The Java layer is critical for integrations with GG for DAA applications.

About Java Layer Logging

The Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) product provides flexibility for logging from the Java layer. The recommended best practice is to use Log4j logging to log from the Java layer. Enabling simple Log4j logging requires the setting of two configuration values in the Java Adapters configuration file.

gg.log=log4j
gg.log.level=INFO

These gg.log settings will result in a Log4j file to be created in the GoldenGate_Home/dirrpt directory that adheres to this naming convention, {GROUPNAME}.log. The supported Log4j log levels are in the following list in order of increasing logging granularity.

Selection of a logging level will include all of the coarser logging levels as well (that is, selection of WARN means that log messages of FATAL, ERROR and WARN will be written to the log file). The Log4j logging can additionally be controlled by separate Log4j properties files. These separate Log4j properties files can be enabled by editing the bootoptions property in the Java Adapter Properties file. These three example Log4j properties files are included with the installation and are included in the classpath:

log4j-default.properties
log4j-debug.properites
log4j-trace.properties

You can modify the bootoptions in any of the files as follows:

javawriter.bootoptions=-Xmx512m -Xms64m -Djava.class.path=.:ggjava/ggjava.jar -Dlog4j.configurationFile=samplelog4j.properties

You can use your own customized Log4j properties file to control logging. The customized Log4j properties file must be available in the Java classpath so that it can be located and loaded by the JVM. The contents of a sample custom Log4j properties file is the following:

javawriter.bootoptions

About SQL Statement Logging

Oracle GoldenGate for Distributed Applications and Analytics now includes the ability to log SQL statements for stage and merge targets into a file. This feature is designed to assist with debugging by providing insight into the execution of SQL statements processed by Replicat.

By default, SQL logging is set to the INFO level. The generated log files are stored in the OGG_DIRLOG directory. Each SQL log entry includes a timestamp to make it easier to trace the execution sequence. Note that SHOWSYNTAX is not supported for coordinated Replicat groups.

This functionality is intended specifically for debugging purposes. It is recommended to use it when you need to verify the SQL syntax generated by Replicat as part of troubleshooting or validation.

Configuring SQL Statement Logging

Parameter file

SHOWSYNTAX _NOCHECKTTY

Property file

For more information about the SHOWSYNTAX parameter, see SHOWSYNTAX in the Reference for Oracle GoldenGate.

Configuring Logging

Oracle GoldenGate Java Adapter Default Logging

Default Logging Setup

Logging is enabled by default for the Oracle GoldenGate for BigData. The logging implementation is log4j. By default, logging is enabled at the info level.

Log File Name

The log output file is created in the standard report directory. The name of the log file includes the replicat group name and has an extension of log.

If the Oracle GoldenGate Replicat process group name is JAVAUE, then the log file name in the report directory is: JAVAUE.log.

Changing Logging Level

To change the recommended log4j logging level, add the configuration shown in the following example to the Java Adapter Properties file:

gg.log.level=error

You can set the gg.log.level to none, error, warn, info, debug, or trace. The default log level is info. Oracle recommends the debug and trace log levels only for troubleshooting as these settings can adversely impact the performance.

Recommended Logging Settings

Oracle recommends that you use log4j logging instead of the JDK default for unified logging for the Java user exit. Using log4j provides unified logging for the Java module when running with the Oracle GoldenGate Replicat process.

Changing to the Recommended Logging Type

To change the recommended log4j logging implementation, add the configuration shown in the following example to the Java Adapter Properties file.

gg.log=log4j
gg.log.level=info

The gg.log level can be set to none, error, warn, info, debug, or trace. The default log level is info. The debug and trace log levels are only recommended for troubleshooting as these settings can adversely affect performance.

The result is that a log file for the Java module will be created in the dirrpt directory with the following naming convention:

*_*`log4j.log`

Therefore if the Oracle GoldenGate Replicat process is called javaue, and the gg.log.level is set to debug, the resulting log file name is:

javaue_debug_log4j.log