1 Understanding the Java Adapter and Oracle GoldenGate for Big Data

This chapter includes the following sections:

1.1 Overview

This section provides an overview of the Oracle GoldenGate Adapters.

1.1.1 Oracle GoldenGate Features

The Oracle GoldenGate Java Adapter integrates with Oracle GoldenGate instances.

The Oracle GoldenGate product enables you to:

  • Captures transactional changes from a source database

  • Sends and queues these changes as a set of database-independent files called the Oracle GoldenGate trail

  • Optionally alters the source data using mapping parameters and functions

  • Applies the transactions in the trail to a target system database

Oracle GoldenGate performs this capture and apply in near real-time across heterogeneous databases, platforms, and operating systems.

1.1.2 Adapter Integration Options

There are two major products which are based on the Oracle GoldenGate Adapter architecture:

  • The Oracle GoldenGate Java Adapter is the overall framework. This product allows you to implement custom code to handle Oracle GoldenGate trail records according to their specific requirements. It comes built-in with Oracle GoldenGate File Writer module that can be used for flat file integration purposes.

  • Oracle GoldenGate for Big Data. The Oracle GoldenGate for Big Data product contains built-in support to write operation data from Oracle GoldenGate trail records into various Big Data targets (HDFS, HBase, Kafka, and Flume). You do not need to write custom code to integrate with Big Data applications. The functionality is separated into handlers that integrate with third party applications and formatters, which transform the data into various formats, such as Avro, JSON, delimited text, and XML. In certain instances, the integration to a third-party tool is proprietary, like the HBase API. In these instances, the formatter exists without an associated handler.

The Oracle GoldenGate Java Adapter and the Oracle GoldenGate for Big Data products have some crossover in functionality. The following list details the major areas of functionality and in which product or products the functionality is included:

  • Read JMS messages and deliver them as an Oracle GoldenGate trail. This feature is included in both Oracle GoldenGate Adapters and Oracle GoldenGate for Big Data products.

  • Read an Oracle GoldenGate trail and deliver transactions to a JMS provider or other messaging system or custom application. This feature is included in both Oracle GoldenGate Adapters and Oracle GoldenGate for Big Data products.

  • Read an Oracle GoldenGate trail and write transactions to a file that can be used by other applications. This feature is only included in the Oracle GoldenGate Adapters product.

  • Read an Oracle GoldenGate trail and write transactions to a Big Data targets. The Big Data integration features are only included in the Oracle GoldenGate for Big Data product. Capturing Transactions to a Trail

Oracle GoldenGate message capture can be used to read messages from a queue and communicate with an Oracle GoldenGate Extract process to generate a trail containing the processed data.

The message capture processing is implemented as a Vendor Access Module (VAM) plug-in to a generic Extract process. A set of properties, rules and external files provide messaging connectivity information and define how messages are parsed and mapped to records in the target Oracle GoldenGatetrail.

Currently this adapter supports capturing JMS text messages. Applying Transactions from a Trail

Oracle GoldenGate Java Adapter delivery can be used to apply transactional changes to targets other than a relational database: for example, ETL tools (DataStage, Ab Initio, Informatica), JMS messaging, Big Data Applications, or custom APIs. There are a variety of options for integration with Oracle GoldenGate:

  • Flat file integration: predominantly for ETL, proprietary or legacy applications, Oracle GoldenGate File Writer can write micro batches to disk to be consumed by tools that expect batch file input. The data is formatted to the specifications of the target application such as delimiter separated values, length delimited values, or binary. Near real-time feeds to these systems are accomplished by decreasing the time window for batch file rollover to minutes or even seconds.

  • Messaging: transactions or operations can be published as messages (for example, in XML) to JMS. The JMS provider is configurable to work with multiple JMS implementation; examples include ActiveMQ, JBoss Messaging, TIBCO, Oracle WebLogic JMS, WebSphere MQ, and others.

  • Java API: custom handlers can be written in Java to process the transaction, operation and metadata changes captured by Oracle GoldenGate on the source system. These custom Java handlers can apply these changes to a third-party Java API exposed by the target system.

  • Big Data integration: writing transaction data from the source trail files into various Big Data targets can be achieved by means of setting configuration properties. The Oracle GoldenGate for Big Data product contains built in Big Data handlers to write to HDFS, HBase, Kafka, and Flume targets.

All four options have been implemented as extensions to the core Oracle GoldenGate product.

  • For the flat file integration, Oracle GoldenGate File Writer provides a user exit library that is dynamically linked into the Oracle GoldenGate Extract process. Configuration is done using a properties file, and no programming is required.

  • For Java integration using either JMS or the Java API, use Oracle GoldenGate for Java.

  • For Big Data integration, you can configure Oracle GoldenGate Replicat to integrate with the Oracle GoldenGate Big Data module. Writing to Big Data targets in various formats can be configured using a set of properties with no programming required.

1.2  Using Oracle GoldenGate Java Adapter Properties

The Oracle GoldenGate user exits and the associated Java Adapters, Big Data Handlers, and Formatters are configured through predefined properties. These properties are stored in a separate properties file called the Adapter Properties file. Oracle GoldenGate functionality requires that the Extract process or Replicat process configuration files must be in the dirprm directory and that configuration files must adhere to the following naming conventions:

Extract or Replicat process name.prm

It is considered to be a best practice that the Adapter Properties files are also located in the dirprm directory and that the Adapter Properties files adhere to one of the following naming conventions:

Extract or Replicat process name.props


Extract or Replicat process name.properties

1.2.1 Values in Property Files

All properties in Oracle GoldenGate Adapter property files are of the form:


1.2.2 Location of Property Files

Sample Oracle GoldenGate Adapter properties files are installed to the AdapterExamples subdirectory of the installation directory. These files should be copied, renamed, and the contents modified as needed and then moved to the dirprm subdirectory.

You must specify each of these property files through parameters or environmental variables as explained below. These settings allow you to change the name or location, but it is recommended that you follow the best practice for naming and location.

The following sample files are included:

  • ffwriter.properties

    This stores the properties for the file writer. It is set with the CUSEREXIT Extract parameter.

  • jmsvam.properties

    This stores properties for the JMS message capture VAM. This is set with the Extract VAM parameter.

  • javaue.properties

    This stores properties for the combined user exit and Java application used for message delivery. It is set through the environmental variable:

    The name and location of the Adapter properties file is resolved by configuration in the either the Extract process properties file or the Replicat process properties file.

    The following section explains how to resolve the name and location of the Adapter Properties file in the Extract process configuration.

    SETENV (GGS_USEREXIT_CONF = "dirprm/javaue.properties")

    Optionally, the Java application properties and native user exit library properties can be in separate files. To do this set GGS_USEREXIT_CONF to the user exit property file and GGS_JAVAUSEREXIT_CONF to the Java application properties file.

    The following explains how to resolve the name and location of the Adapter Properties file in the Replicat process.

    TARGETDB LIBFILE libggjava.so SET property=dirprm/javaue.properties

1.2.3 Using Comments in the Property File

Comments can be entered in the properties file with the # prefix at the beginning of the line. For example:

# This is a property comment 

Properties themselves can also be commented. This allows testing configurations without losing previous property settings.

1.2.4 Variables in Property Names

Some properties have a variable in the property name. This allows identification of properties that are to be applied only in certain instances.

For example, you can declare more than one file writer using goldengate.flatfilewriter.writers property and then use the name of the file writer to set the properties differently:

  1. Declare two file writers named writer and writer2:
  2. Specify the properties for each of the file writers:

1.3 Oracle GoldenGate Documentation

For information on installing and configuring the core Oracle GoldenGate software for use with the Oracle GoldenGate adapter products, see the Oracle GoldenGate documentation:

  • Installation and Setup guides: There is one such guide for each database that is supported by Oracle GoldenGate. It contains system requirements, pre-installation and post-installation procedures, installation instructions, and other system-specific information for installing the Oracle GoldenGate replication solution.

  • : Explains how to plan for, configure, and implement the Oracle GoldenGate replication solution on the Windows and UNIX platforms.

  • : Contains detailed information about Oracle GoldenGate parameters, commands, and functions for the Windows and UNIX platforms.