1.1 Understanding Oracle GoldenGate for Distributed Applications and Analytics

This section describes the concepts and basic structure of the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA).

Watch this video for an introduction to Oracle GoldenGate Microservices: Introduction to GoldenGate Microservices

• Understanding Oracle GoldenGate for Distributed Applications and Analytics
  

1.1.1 Understanding Oracle GoldenGate for Distributed Applications and Analytics

The Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) integrates with Oracle GoldenGate instances.

The Oracle GoldenGate product enables you to:

• Capture transactional changes from a source database.
• Sends and queues these changes as a set of database-independent files called the Oracle GoldenGate trail.
• Optionally alters the source data using mapping parameters and functions.
• Applies the transactions in the trail to a target system database.

Oracle GoldenGate performs this capture and apply in near real-time across heterogeneous databases, platforms, and operating systems.

• Delivery Configuration Options
  
• Adapter Integration Options
  
• Monitoring Performance
  

1.1.1.1 Delivery Configuration Options

The Java delivery module is loaded by the GoldenGate Replicat process, which is configured using the Replicat parameter file. Upon loading, the Java Delivery module is subsequently configured based on the configuration present in the Adapter Properties file. Application behavior can be customized by:

• Editing the property files; for example to:

  • Set target types, host names, port numbers, output file names, JMS connection settings;

  • Turn on/off debug-level logging, and so on.

  • Identify which message format should be used.

• Records can be custom formatted by:

  • Setting properties for the pre-existing format process (for fixed-length or field-delimited message formats, XML, JSON, or Avro formats);

  • Customizing message templates, using the Velocity template macro language;

  • (Optional) Writing custom Java code.

• (Optional) Writing custom Java code to provide custom handling of transactions and operations, do filtering, or implementing custom message formats.

There are existing implementations (handlers) for sending messages using JMS and for writing out files to disk. For Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) targets, there are built in integration handlers to write to supported databases.

There are several predefined message formats for sending the messages (for example, XML or field-delimited); or custom formats can be implemented using templates. Each handler has documentation that describes its configuration properties; for example, a file name can be specified for a file writer, and a JMS queue name can be specified for the JMS handler. Some properties apply to more than one handler; for example, the same message format can be used for JMS and files.

1.1.1.2 Adapter Integration Options

There are two major products which are based on the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) architecture:

• The Oracle GoldenGate Java Adapter is the overall framework. This product allows you to implement custom code to handle Oracle GoldenGate trail records according to their specific requirements. It comes built-in with Oracle GoldenGate File Writer module that can be used for flat file integration purposes.

• GG for DAA. The GG for DAA product contains built-in support to write operation data from Oracle GoldenGate trail records into various GG for DAA targets (such as, HDFS, HBase, Kafka, Flume, JDBC, Cassandra, and MongoDB). You do not need to write custom code to integrate with GG for DAA applications. The functionality is separated into handlers that integrate with third party applications and formatters, which transform the data into various formats, such as Avro, JSON, delimited text, and XML. In certain instances, the integration to a third-party tool is proprietary, like the HBase API. In these instances, the formatter exists without an associated handler.

The Oracle GoldenGate Java Adapter and the GG for DAA products have some crossover in functionality so the handler exists without an associated formatter. The following list details the major areas of functionality and in which product or products the functionality is included:

• Read JMS messages and deliver them as an Oracle GoldenGate trail. This feature is included in GG for DAA.

• Read an Oracle GoldenGate trail and deliver transactions to a JMS provider or other messaging system or custom application. This feature is included in GG for DAA products.

• Read an Oracle GoldenGate trail and write transactions to a file that can be used by other applications. This feature is only included in GG for DAA.

• Read an Oracle GoldenGate trail and write transactions to a GG for DAA targets. The GG for DAA integration features are only included in GG for DAA product.

• Capturing Transactions to a Trail
  
• Applying Transactions from a Trail
  

1.1.1.2.1 Capturing Transactions to a Trail

Oracle GoldenGate message capture can be used to read messages from a queue and communicate with an Oracle GoldenGate Extract process to generate a trail containing the processed data.

The message capture processing is implemented as a Vendor Access Module (VAM) plug-in to a generic Extract process. A set of properties, rules and external files provide messaging connectivity information and define how messages are parsed and mapped to records in the target Oracle GoldenGate trail.

Currently this adapter supports capturing JMS text messages.

1.1.1.2.2 Applying Transactions from a Trail

Oracle GoldenGate Java Adapter delivery can be used to apply transactional changes to targets other than a relational database: for example, ETL tools (DataStage, Ab Initio, Informatica), JMS messaging, Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) Applications, or custom APIs. There are a variety of options for integration with Oracle GoldenGate:

• Flat file integration: predominantly for ETL, proprietary or legacy applications, Oracle GoldenGate File Writer can write micro batches to disk to be consumed by tools that expect batch file input. The data is formatted to the specifications of the target application such as delimiter separated values, length delimited values, or binary. Near real-time feeds to these systems are accomplished by decreasing the time window for batch file rollover to minutes or even seconds.

• Messaging: transactions or operations can be published as messages (for example, in XML) to JMS. The JMS provider is configurable to work with multiple JMS implementation; examples include ActiveMQ, JBoss Messaging, TIBCO, Oracle WebLogic JMS, WebSphere MQ, and others.

• Java API: custom handlers can be written in Java to process the transaction, operation and metadata changes captured by Oracle GoldenGate on the source system. These custom Java handlers can apply these changes to a third-party Java API exposed by the target system.

• GG for DAA integration: writing transaction data from the source trail files into various GG for DAA targets can be achieved by means of setting configuration properties. The GG for DAA product contains built in GG for DAA handlers to write to HDFS, HBase, Kafka, and Flume targets.

All four options have been implemented as extensions to the core Oracle GoldenGate product.

• For Java integration using either JMS or the Java API, use Oracle GoldenGate for Java.

• For GG for DAA integration, you can configure Oracle GoldenGate Replicat to integrate with the GG for DAA module. Writing to GG for DAA targets in various formats can be configured using a set of properties with no programming required.

1.1.1.3 Monitoring Performance

For more information about monitoring the performance, see Monitor Performance from the Performance Metrics Service in Using Oracle GoldenGate Microservices Architecture.