1.1 Understanding Oracle GoldenGate for Big Data

This section describes the concepts and basic structure of the Oracle GoldenGate for Big Data.

Watch this video for an introduction to Oracle GoldenGate Microservices: Introduction to GoldenGate 21c Microservices

• Understanding the Java Adapter and Oracle GoldenGate for Big Data
  

1.1.1 Understanding the Java Adapter and Oracle GoldenGate for Big Data

The Oracle GoldenGate Java Adapter integrates with Oracle GoldenGate instances.

The Oracle GoldenGate product enables you to:

• Capture transactional changes from a source database.
• Sends and queues these changes as a set of database-independent files called the Oracle GoldenGate trail.
• Optionally alters the source data using mapping parameters and functions.
• Applies the transactions in the trail to a target system database.

Oracle GoldenGate performs this capture and apply in near real-time across heterogeneous databases, platforms, and operating systems.

• Oracle GoldenGate VAM Message Capture
  
• Oracle GoldenGate Java Delivery
  
• Delivery Configuration Options
  
• Adapter Integration Options
  
•  Using Oracle GoldenGate Java Adapter Properties
  
• Monitoring Performance
  

1.1.1.1 Oracle GoldenGate VAM Message Capture

Oracle GoldenGate VAM Message Capture only works with the Oracle GoldenGate Extract process. Oracle GoldenGate message capture connects to JMS messaging to parse messages and send them through a VAM interface to an Oracle GoldenGate Extract process that builds an Oracle GoldenGate trail of message data. This allows JMS messages to be delivered to an Oracle GoldenGate system running for a target database. Java 8 is a required dependency for Oracle GoldenGate VAM Message Capture.

Using Oracle GoldenGate JMS message capture requires the dynamically linked shared VAM library that is attached to the Oracle GoldenGate Extract process.

• Message Capture Configuration Options
  
• Typical Configuration
  

1.1.1.1.1 Message Capture Configuration Options

The options for configuring the three parts of message capture are:

• Message connectivity: Values in the property file set connection properties such as the Java classpath for the JMS client, the JMS source destination name, JNDI connection properties, and security information.

• Parsing: Values in the property file set parsing rules for fixed width, comma delimited, or XML messages. This includes settings such as the delimiter to be used, values for the beginning and end of transactions and the date format.

• VAM interface: Parameters that identify the VAM, dll, or so library and a property file are set for the Oracle GoldenGate core Extract process.

1.1.1.1.2 Typical Configuration

The following diagram shows a typical configuration for capturing JMS messages.

Figure 1-1 Configuration for JMS Message Capture

Description of "Figure 1-1 Configuration for JMS Message Capture"

In this configuration, JMS messages are picked up by the Oracle GoldenGate for Big Data JMS Handler and transferred using the adapter's message capture VAM to an Extract process. The Extract writes the data to a trail which is sent over the network by a Data Pump Extract to an Oracle GoldenGate target instance. The target Replicat then uses the trail to update the target database.

1.1.1.2 Oracle GoldenGate Java Delivery

Through the Oracle GoldenGate Java API, transactional data captured by Oracle GoldenGate can be delivered to targets other than a relational database, such as a JMS (Java Message Service), files written to disk, streaming data to a Big Data application, or integration with a custom application Java API. Oracle GoldenGate Java Delivery can work with either an Extract or Replicat process. Using the Oracle GoldenGate Replicat process is considered the best practice. Oracle GoldenGate Java Delivery requires Java 8 as a dependency.

Oracle GoldenGate for Java provides the ability to execute Java code from the Oracle GoldenGate Replicat process. Using Oracle GoldenGate for Java requires the following conditions to be met:

• A dynamically linked or shared library, implemented in C/C++, integrating an extension module of Oracle GoldenGate Replicat process.

• A set of Java libraries (JARs), which comprise the Oracle GoldenGate Java API. This Java framework communicates with the Replicat through the Java Native Interface (JNI).

• Java 8 must be installed and accessible on the machine hosting the Oracle GoldenGate Java Delivery process or processes. Environmental variables must be correctly set to resolve Java and its associated libraries.

Figure 1-2 Configuration for Delivering JMS Messages

Description of "Figure 1-2 Configuration for Delivering JMS Messages"

1.1.1.3 Delivery Configuration Options

The Java delivery module is loaded by the GoldenGate Replicat process, which is configured using the Replicat parameter file. Upon loading, the Java Delivery module is subsequently configured based on the configuration present in the Adapter Properties file. Application behavior can be customized by:

• Editing the property files; for example to:

  • Set target types, host names, port numbers, output file names, JMS connection settings;

  • Turn on/off debug-level logging, and so on.

  • Identify which message format should be used.

• Records can be custom formatted by:

  • Setting properties for the pre-existing format process (for fixed-length or field-delimited message formats, XML, JSON, or Avro formats);

  • Customizing message templates, using the Velocity template macro language;

  • (Optional) Writing custom Java code.

• (Optional) Writing custom Java code to provide custom handling of transactions and operations, do filtering, or implementing custom message formats.

There are existing implementations (handlers) for sending messages using JMS and for writing out files to disk. For Big Data targets, there are built in integration handlers to write to supported databases.

There are several predefined message formats for sending the messages (for example, XML or field-delimited); or custom formats can be implemented using templates. Each handler has documentation that describes its configuration properties; for example, a file name can be specified for a file writer, and a JMS queue name can be specified for the JMS handler. Some properties apply to more than one handler; for example, the same message format can be used for JMS and files.

1.1.1.4 Adapter Integration Options

There are two major products which are based on the Oracle GoldenGate for Big Data architecture:

• The Oracle GoldenGate Java Adapter is the overall framework. This product allows you to implement custom code to handle Oracle GoldenGate trail records according to their specific requirements. It comes built-in with Oracle GoldenGate File Writer module that can be used for flat file integration purposes.

• Oracle GoldenGate for Big Data. The Oracle GoldenGate for Big Data product contains built-in support to write operation data from Oracle GoldenGate trail records into various Big Data targets (such as, HDFS, HBase, Kafka, Flume, JDBC, Cassandra, and MongoDB). You do not need to write custom code to integrate with Big Data applications. The functionality is separated into handlers that integrate with third party applications and formatters, which transform the data into various formats, such as Avro, JSON, delimited text, and XML. In certain instances, the integration to a third-party tool is proprietary, like the HBase API. In these instances, the formatter exists without an associated handler.

The Oracle GoldenGate Java Adapter and the Oracle GoldenGate for Big Data products have some crossover in functionality so the handler exists without an associated formatter. The following list details the major areas of functionality and in which product or products the functionality is included:

• Read JMS messages and deliver them as an Oracle GoldenGate trail. This feature is included in Oracle GoldenGate for Big Data.

• Read an Oracle GoldenGate trail and deliver transactions to a JMS provider or other messaging system or custom application. This feature is included in Oracle GoldenGate for Big Data products.

• Read an Oracle GoldenGate trail and write transactions to a file that can be used by other applications. This feature is only included in Oracle GoldenGate for Big Data.

• Read an Oracle GoldenGate trail and write transactions to a Big Data targets. The Big Data integration features are only included in Oracle GoldenGate for Big Data product.

• Capturing Transactions to a Trail
  
• Applying Transactions from a Trail
  

1.1.1.4.1 Capturing Transactions to a Trail

Oracle GoldenGate message capture can be used to read messages from a queue and communicate with an Oracle GoldenGate Extract process to generate a trail containing the processed data.

The message capture processing is implemented as a Vendor Access Module (VAM) plug-in to a generic Extract process. A set of properties, rules and external files provide messaging connectivity information and define how messages are parsed and mapped to records in the target Oracle GoldenGate trail.

Currently this adapter supports capturing JMS text messages.

1.1.1.4.2 Applying Transactions from a Trail

Oracle GoldenGate Java Adapter delivery can be used to apply transactional changes to targets other than a relational database: for example, ETL tools (DataStage, Ab Initio, Informatica), JMS messaging, Big Data Applications, or custom APIs. There are a variety of options for integration with Oracle GoldenGate:

• Flat file integration: predominantly for ETL, proprietary or legacy applications, Oracle GoldenGate File Writer can write micro batches to disk to be consumed by tools that expect batch file input. The data is formatted to the specifications of the target application such as delimiter separated values, length delimited values, or binary. Near real-time feeds to these systems are accomplished by decreasing the time window for batch file rollover to minutes or even seconds.

• Messaging: transactions or operations can be published as messages (for example, in XML) to JMS. The JMS provider is configurable to work with multiple JMS implementation; examples include ActiveMQ, JBoss Messaging, TIBCO, Oracle WebLogic JMS, WebSphere MQ, and others.

• Java API: custom handlers can be written in Java to process the transaction, operation and metadata changes captured by Oracle GoldenGate on the source system. These custom Java handlers can apply these changes to a third-party Java API exposed by the target system.

• Big Data integration: writing transaction data from the source trail files into various Big Data targets can be achieved by means of setting configuration properties. The Oracle GoldenGate for Big Data product contains built in Big Data handlers to write to HDFS, HBase, Kafka, and Flume targets.

All four options have been implemented as extensions to the core Oracle GoldenGate product.

• For Java integration using either JMS or the Java API, use Oracle GoldenGate for Java.

• For Big Data integration, you can configure Oracle GoldenGate Replicat to integrate with the Oracle GoldenGate Big Data module. Writing to Big Data targets in various formats can be configured using a set of properties with no programming required.

1.1.1.5  Using Oracle GoldenGate Java Adapter Properties

The Oracle GoldenGate Java Adapters, Big Data Handlers, and formatters are configured through predefined properties. These properties are stored in a separate properties file called the Adapter Properties file. Oracle GoldenGate functionality requires that the Replicat process configuration files must be in the dirprm directory and that configuration files must adhere to the following naming conventions:

Replicat process name.prm

It is considered to be a best practice that the Adapter Properties files are also located in the dirprm directory and that the Adapter Properties files adhere to one of the following naming conventions:

Replicat process name.props

or

Replicat process name.properties

• Values in Property Files
  
• Location of Property Files
  
• Using Comments in the Property File
  
• Variables in Property Names
  

1.1.1.5.1 Values in Property Files

All properties in Oracle GoldenGate for Big Data property files are of the form:

property.name=value 

1.1.1.5.2 Location of Property Files

Sample Oracle GoldenGate for Big Data properties files are installed to the AdapterExamples subdirectory of the installation directory. These files should be copied, renamed, and the contents modified as needed and then moved to the dirprm subdirectory.

You must specify each of these property files through parameters or environmental variables as explained below. These settings allow you to change the name or location, but it is recommended that you follow the best practice for naming and location.

The following sample files are included:

• ffwriter.properties

  This stores the properties for the file writer. It is set with the CUSEREXIT parameter.

• jmsvam.properties

  This stores properties for the JMS message capture VAM. This is set with the Extract VAM parameter.

• javaue.properties

  This stores properties for the Java application used for message delivery. It is set through the environmental variable:

  The name and location of the Adapter properties file is resolved by configuration in the Replicat process properties file.

  The following explains how to resolve the name and location of the Adapter Properties file in the Replicat process.

  TARGETDB LIBFILE libggjava.so SET property=dirprm/javaue.properties

1.1.1.5.3 Using Comments in the Property File

Comments can be entered in the properties file with the # prefix at the beginning of the line. For example:

# This is a property comment 
some.property=value 

Properties themselves can also be commented. This allows testing configurations without losing previous property settings.

1.1.1.5.4 Variables in Property Names

Some properties have a variable in the property name. This allows identification of properties that are to be applied only in certain instances.

For example, you can declare more than one file writer using goldengate.flatfilewriter.writers property and then use the name of the file writer to set the properties differently:

1. Declare two file writers named writer and writer2:

   goldengate.flatfilewriter.writers=writer,writer2 
   

2. Specify the properties for each of the file writers:

   writer.mode=dsv 
   writer.files.onepertable=true
   writer2.mode=ldv 
   writer2.files.onpertable=false

1.1.1.6 Monitoring Performance

For more information about monitoring the performance, see Monitor Performance from the Performance Metrics Service in Using Oracle GoldenGate Microservices Architecture.