1 Overview

The following is an example of a Replicat process properties file for Java Adapter.

REPLICAT hdfs
TARGETDB LIBFILE libggjava.so SET property=dirprm/hdfs.properties 
--SOURCEDEFS ./dirdef/dbo.def 
DDL INCLUDE ALL
GROUPTRANSOPS 1000
MAPEXCLUDE dbo.excludetable
MAP dbo.*, TARGET dbo.*;

The following is explanation of the Replicat configuration entries:

REPLICAT hdfs - The name of the Replicat process.

TARGETDB LIBFILE libggjava.so SET property=dirprm/hdfs.properties - Names the target database as you exit libggjava.so and sets the Java Adapters Property file to dirprm/hdfs.properties

--SOURCEDEFS ./dirdef/dbo.def - Sets a source database definitions file. Commented out because Oracle GoldenGate 12.2.0.1 trail files provide metadata in trail.

GROUPTRANSOPS 1000 - To group 1000 transactions from the source trail files into a single target transaction. This is the default and improves the performance of Big Data integrations.

MAPEXCLUDE dbo.excludetable - To identify tables to exclude.

MAP dbo.*, TARGET dbo.*; - Shows the mapping of input to output tables.

The command to add and start the Replicat process in ggsci is the following:

ADD REPLICAT hdfs, EXTTRAIL ./dirdat/gg
START hdfs

The Replicat process provides the Replicat configuration property GROUPTRANSOPS to control transaction grouping. By default, the Replicat process implements transaction grouping of 1000 source transactions into a single target transaction. If you want to turn off transaction grouping then the GROUPTRANSOPS Replicat property should be set to 1.

CHECKPOINTTABLE and NODBCHECKPOINT are not applicable for Java Delivery with Replicat. Beside Replicat checkpoint file (.cpr), additional checkpoint file (dirchk/<group>.cpj) will be created that contains information similar to CHECKPOINTTABLE in Replicat for RDBMS.

The following Replicat features are not supported in this release:

• BATCHSQL

• SQLEXEC

• Stored procedure

• Conflict resolution and detection (CDR)

• REPERROR

The Oracle GoldenGate Replicat process supports mapping functionality to custom target schemas. This functionality is not available using the Oracle GoldenGate Extract process. You must use the Metadata Provider functionality to define a target schema or schemas and then use the standard Replicat mapping syntax in the Replicat configuration file to define the mapping. Refer to the Oracle GoldenGate Replicat documentation to understand the Replicat mapping syntax in the Replication configuration file. For instructions on setting up the Metadata Provider, see Using the Metadata Provider.

The following

EXTRACT hdfs
discardfile ./dirrpt/avro1.dsc, purge 
--SOURCEDEFS ./dirdef/dbo.def
CUSEREXIT libjavaue.so CUSEREXIT PASSTHRU, INCLUDEUPDATEBEFORES, PARAMS "dirprm/hdfs.props" 
GETUPDATEBEFORES 
TABLE dbo.*;

The following is explanation of the Replicat configuration entries:

EXTRACT hdfs - The Extract process name.

discardfile ./dirrpt/avro1.dsc, purge - Set the discard file

--SOURCEDEFS ./dirdef/dbo.def - Source definitions are not required for 12.2 trial files.

CUSEREXIT libjavaue.so CUSEREXIT PASSTHRU, INCLUDEUPDATEBEFORES, PARAMS "dirprm/hdfs.props" - Set you exit shared library, and point to the Java Adapter Properties file

GETUPDATEBEFORES - Get update before images.

TABLE dbo.*; - Select which tables to replicate or exclude to filter.

ADD EXTRACT hdfs, EXTTRAILSOURCE ./dirdat/gg
START hdfs

The Extract process provides no functionality for transaction grouping. However, transaction grouping is still possible when integrating Java Delivery with the Extract process. The Java Delivery layer enables transaction grouping with configuration in the Java Adapter properties file.

1. gg.handler.name.mode

   To enable grouping, the value of this property must be set to tx.

2. gg.handler.name.maxGroupSize

   Controls the maximum number of operations that can be held by an operation group - irrespective of whether the operation group holds operations from a single transaction or multiple transactions.

   The operation group will send a transaction commit and end the group as soon as this number of operations is reached. This property leads to splitting of transactions across multiple operation groups.

3. gg.handler.name.minGroupSize

   This is the minimum number of operations that must exist in a group before the group can end.

   This property helps to avoid groups that are too small by grouping multiple small transactions into one operation group so that it can be more efficiently processed.

   Note:

   maxGroupSize should always be greater than or equal to minGroupSize; that is, maxGroupSize >= minGroupSize.

Oracle GoldenGate for Big Data integrations leverage the Java Delivery functionality described in the Oracle GoldenGate Application Adapters Guide. In this setup, either a Oracle GoldenGate Replicat or Extract process loads a user exit shared library. This shared library then loads a Java virtual machine to thereby interface with targets providing a Java interface. So the flow of data is as follows:

Extract Process > User Exit > Java Layer

or

Replicat Process >User Exit > Java Layer

It is important that all layers log correctly so that users can review the logs to troubleshoot new installations and integrations. Additionally, if a customer has a problem that requires contacting Oracle Support, the log files are a key piece of information to be provided to Oracle Support so that the problem can be efficiently resolved.

A running Replicat or Extract process creates or appends log files into the <GG Home>/dirrpt directory that adheres to the following naming convention: <Replicat or Extract process name>.rpt. If a problem is encountered when deploying a new Oracle GoldenGate process, this is likely the first log file to examine for problems. The Java layer provides much of the heavy lifting for integrations with Big Data applications. Therefore are many things that can go wrong in the Java layer when performing the initial setup of a Oracle GoldenGate Big Data integration. You therefore need to understand how to control logging in the Java layer.

The Oracle GoldenGate for Big Data product provides flexibility for logging from the Java layer. The recommended best practice is to use Log4j logging to log from the Java layer. Enabling simple Log4j logging requires the setting of two configuration values in the Java Adapters configuration file.

gg.log=log4j
gg.log.level=INFO

These gg.log settings will result in a Log4j file to be created in the GoldenGate_Home/dirrpt directory that adheres to this naming convention, Replicat or Extract process name_log level_log4j.log. The supported Log4j log levels are in the following list in order of increasing logging granularity.

• OFF

• FATAL

• ERROR

• WARN

• INFO

• DEBUG

• TRACE

Selection of a logging level will include all of the coarser logging levels as well (that is, selection of WARN means that log messages of FATAL, ERROR and WARN will be written to the log file). The Log4j logging can additionally be controlled by separate Log4j properties files. These separate Log4j properties files can be enabled by editing the bootoptions property in the Java Adapter Properties file. Three example Log4j properties files are included with the installation and, are included in the classpath:

log4j-default.properties
log4j-debug.properites
log4j-trace.properties

Any one of these files can be modifying the bootoptions as follows:

javawriter.bootoptions=-Xmx512m -Xms64m -Djava.class.path=.:ggjava/ggjava.jar -Dlog4j.configuration=samplelog4j.properties

You can use their own customized Log4j properties file to control logging. The customized Log4j properties file must be available in the Java classpath so that it can be located and loaded by the JVM. The contents of a sample custom Log4j properties file is the following:

# Root logger option 
log4j.rootLogger=INFO, file 
 
# Direct log messages to a log file 
log4j.appender.file=org.apache.log4j.RollingFileAppender 
 
log4j.appender.file.File=sample.log 
log4j.appender.file.MaxFileSize=1GB 
log4j.appender.file.MaxBackupIndex=10 
log4j.appender.file.layout=org.apache.log4j.PatternLayout 
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

There are two important requirements when you use a custom Log4j properties file. First, the path to the custom Log4j properties file must be included in the javawriter.bootoptions property. Logging initializes immediately when the JVM is initialized while the contents of the gg.classpath property is actually appended to the classloader after the logging is initialized. Second, the classpath to correctly load a properties file must be the directory containing the properties file without wildcards appended.