Go to main content
1/19
Contents
Title and Copyright Information
Preface
Audience
Documentation Accessibility
Related Information
Conventions
1
Introduction to GoldenGate for Big Data
1.1
Introduction
1.2
Understanding What is Supported
1.2.1
Verifying Certification and System Requirements
1.2.2
Understanding Handler Compatibility
1.2.2.1
HDFS Handler
1.2.2.2
HBase Handler
1.2.2.3
Flume Handler
1.2.2.4
Kafka Handler
1.2.2.5
Cassandra Handler
1.2.2.6
MongoDB Handler
1.2.2.7
JBDC Handler
1.2.3
What are the Additional Support Considerations?
1.3
Setting Up Oracle GoldenGate for Big Data
1.3.1
Java Environment Setup
1.3.2
Properties Files
1.3.3
Transaction Grouping
1.4
Configuring GoldenGate for Big Data
1.4.1
Running with Replicat
1.4.1.1
Configuring Replicat
1.4.1.2
Adding the Replicat Process
1.4.1.3
Replicat Grouping
1.4.1.4
Replicat Checkpointing
1.4.1.5
Unsupported Replicat Features
1.4.1.6
Mapping Functionality
1.4.2
Logging
1.4.2.1
Replicat Process Logging
1.4.2.2
Java Layer Logging
1.4.3
Metadata Change Events
1.4.4
Configuration Property CDATA[] Wrapping
1.4.5
Using Regular Expression Search and Replace
1.4.5.1
Using Schema Data Replace
1.4.5.2
Using Content Data Replace
1.4.6
Scaling Oracle GoldenGate for Big Data Delivery
1.4.7
Using Identities in Oracle GoldenGate Credential Store
1.4.7.1
Creating a Credential Store
1.4.7.2
Adding Users to a Credential Store
1.4.7.3
Configuring Properties to Access the Credential Store
2
Using the HDFS Handler
2.1
Overview
2.2
Writing into HDFS in SequenceFile Format
2.2.1
Integrating with Hive
2.2.2
Understanding the Data Format
2.2.3
Setting Up and Running the HDFS Handler
2.2.3.1
Classpath Configuration
2.2.3.2
HDFS Handler Configuration
2.2.3.3
Sample Configuration
2.2.3.4
Performance Considerations
2.2.3.5
Security
2.3
Writing in HDFS in Avro Object Container File Format
2.4
Metadata Change Events
2.5
Partitioning
2.6
Additional Considerations
2.7
Best Practices
2.8
Troubleshooting the HDFS Handler
2.8.1
Java Classpath
2.8.2
HDFS Connection Properties
2.8.3
Handler and Formatter Configuration
3
Using the HBase Handler
3.1
Overview
3.2
Detailed Functionality
3.3
Setting Up and Running the HBase Handler
3.3.1
Classpath Configuration
3.3.2
HBase Handler Configuration
3.3.3
Sample Configuration
3.3.4
Performance Considerations
3.3.5
Security
3.4
Metadata Change Events
3.5
Additional Considerations
3.6
Troubleshooting the HBase Handler
3.6.1
Java Classpath
3.6.2
HBase Connection Properties
3.6.3
Logging of Handler Configuration
4
Using the Flume Handler
4.1
Overview
4.2
Setting Up and Running the Flume Handler
4.2.1
Classpath Configuration
4.2.2
Flume Handler Configuration
4.2.3
Sample Configuration
4.3
Data Mapping of Operations to Flume Events
4.3.1
Operation Mode
4.3.2
Transaction Mode and EventMapsTo Operation
4.3.3
Transaction Mode and EventMapsTo Transaction
4.4
Performance Considerations
4.5
Metadata Change Events
4.6
Example Flume Source Configuration
4.6.1
Avro Flume Source
4.6.2
Thrift Flume Source
4.7
Advanced Features
4.7.1
Schema Propagation
4.7.2
Security
4.7.3
Fail Over Functionality
4.7.4
Load Balancing Functionality
4.8
Troubleshooting the Flume Handler
4.8.1
Java Classpath
4.8.2
Flume Flow Control Issues
4.8.3
Flume Agent Configuration File Not Found
4.8.4
Flume Connection Exception
4.8.5
Other Failures
5
Using the Kafka Handler
5.1
Overview
5.2
Detailed Functionality
5.3
Setting Up and Running the Kafka Handler
5.3.1
Classpath Configuration
5.3.2
Kafka Handler Configuration
5.3.3
Java Adapter Properties File
5.3.4
Kafka Producer Configuration File
5.4
Schema Propagation
5.5
Performance Considerations
5.6
Security
5.7
Metadata Change Events
5.8
Snappy Considerations
5.9
Troubleshooting
5.9.1
Verify the Kafka Setup
5.9.2
Classpath Issues
5.9.3
Invalid Kafka Version
5.9.4
Kafka Producer Properties File Not Found
5.9.5
Kafka Connection Problem
6
Using the Cassandra Handler
6.1
Overview
6.2
Detailed Functionality
6.2.1
Cassandra Data Types
6.2.2
Catalog, Schema, Table, and Column Name Mapping
6.2.3
DDL Functionality
6.2.3.1
Keyspaces
6.2.3.2
Tables
6.2.3.3
Add Column Functionality
6.2.3.4
Drop Column Functionality
6.2.4
Operation Processing
6.2.5
Compressed Updates vs. Full Image Updates
6.2.6
Primary Key Updates
6.3
Setting Up and Running the Cassandra Handler
6.3.1
Cassandra Handler Configuration
6.3.2
Sample Configuration
6.3.3
Security
6.4
Automated DDL Handling
6.4.1
Table Check and Reconciliation Process
6.5
Performance Considerations
6.6
Additional Considerations
6.7
Troubleshooting
6.7.1
Java Classpath
6.7.2
Logging
6.7.3
Write Timeout Exception
7
Using the Java Database Connectivity Handler
7.1
Overview
7.2
Detailed Functionality
7.2.1
Single Operation Mode
7.2.2
Oracle Database Data Types
7.2.3
MySQL Database Data Types
7.2.4
Netezza Database Data Types
7.2.5
Redshift Database Data Types
7.3
Setting Up and Running the JDBC Handler
7.3.1
Java Classpath
7.3.2
Handler Configuration
7.3.3
Statement Caching
7.3.4
Setting Up Error Handling
7.4
Sample Configurations
7.4.1
Sample Oracle Database Target
7.4.2
Sample Oracle Database Target with JDBC Metadata Provider
7.4.3
Sample MySQL Database Target
7.4.4
Sample MySQL Database Target with JDBC Metadata Provider
8
Using the MongoDB Handler
8.1
Overview
8.2
Detailed Functionality
8.2.1
Document Key Column
8.2.2
Primary Key Update Operation
8.2.3
MongoDB Trail Data Types
8.3
Setting Up and Running the MongoDB Handler
8.3.1
Classpath Configuration
8.3.2
MongoDB Handler Configuration
8.3.3
Connecting and Authenticating
8.3.4
Using Bulk Write
8.3.5
Using Write Concern
8.3.6
Using Three-Part Table Names
8.3.7
Using Undo Handling
8.4
Sample Configuration
9
Using the Pluggable Formatters
9.1
Operation vs. Row Based Formatting
9.1.1
Operation Formatters
9.1.2
Row Formatters
9.1.3
Table Row or Column Value States
9.2
Delimited Text Formatter
9.2.1
Message Formatting Details
9.2.2
Sample Formatted Messages
9.2.2.1
Sample Insert Message
9.2.2.2
Sample Update Message
9.2.2.3
Sample Delete Message
9.2.2.4
Sample Truncate Message
9.2.3
Additional Considerations
9.2.3.1
Primary Key Updates
9.2.3.2
Data Consolidation
9.2.4
Output Format Summary Log
9.2.5
Delimited Text Format Configuration
9.2.6
Sample Configuration
9.2.7
Metadata Change Events
9.3
JSON Formatter
9.3.1
Operation Metadata Formatting Details
9.3.2
Operation Data Formatting Details
9.3.3
Row Data Formatting Details
9.3.4
Sample JSON Messages
9.3.4.1
Sample Operation Modeled JSON Messages
9.3.4.2
Sample Flattened Operation Modeled JSON Messages
9.3.4.3
Sample Row Modeled JSON Messages
9.3.4.4
Sample Primary Key Output JSON Message
9.3.5
JSON Schemas
9.3.6
JSON Formatter Configuration
9.3.7
Sample Configuration
9.3.8
Metadata Change Events
9.3.9
JSON Primary Key Updates
9.3.10
Integrating Oracle Stream Analytics
9.4
Avro Formatter
9.4.1
Avro Row Formatter
9.4.1.1
Operation Metadata Formatting Details
9.4.1.2
Operation Data Formatting Details
9.4.1.3
Sample Avro Row Messages
9.4.1.3.1
Sample Insert Message
9.4.1.3.2
Sample Update Message
9.4.1.3.3
Sample Delete Message
9.4.1.3.4
Sample Truncate Message
9.4.1.4
Avro Schemas
9.4.1.5
Avro Row Configuration
9.4.1.6
Sample Configuration
9.4.1.7
Metadata Change Events
9.4.1.8
Special Considerations
9.4.1.8.1
Troubleshooting
9.4.1.8.2
Primary Key Updates
9.4.1.8.3
Generic Wrapper Functionality
9.4.2
Avro Operation Formatter
9.4.2.1
Operation Metadata Formatting Details
9.4.2.2
Operation Data Formatting Details
9.4.2.3
Sample Avro Operation Messages
9.4.2.3.1
Sample Insert Message
9.4.2.3.2
Sample Update Message
9.4.2.3.3
Sample Delete Message
9.4.2.3.4
Sample Truncate Message
9.4.2.4
Avro Schema
9.4.2.5
Avro Operation Formatter Configuration
9.4.2.6
Sample Configuration
9.4.2.7
Metadata Change Events
9.4.2.8
Special Considerations
9.4.2.8.1
Troubleshooting
9.4.2.8.2
Primary Key Updates
9.4.2.8.3
Generic Wrapper Message
9.4.3
Avro Object Container File Formatter
9.4.3.1
Avro OCF Formatter Configuration
9.5
XML Formatter
9.5.1
Message Formatting Details
9.5.2
Sample XML Messages
9.5.2.1
Sample Insert Message
9.5.2.2
Sample Update Message
9.5.2.3
Sample Delete Message
9.5.2.4
Sample Truncate Message
9.5.3
XML Schema
9.5.4
XML Configuration
9.5.5
Sample Configuration
9.5.6
Metadata Change Events
9.5.7
Primary Key Updates
10
Using the Metadata Provider
10.1
About the Metadata Provider
10.2
Avro Metadata Provider
10.2.1
Detailed Functionality
10.2.2
Runtime Prerequisites
10.2.3
Classpath Configuration
10.2.4
Avro Metadata Provider Configuration
10.2.5
Sample Configuration
10.2.6
Metadata Change Event
10.2.7
Limitations
10.2.8
Troubleshooting
10.2.8.1
Invalid Schema Files Location
10.2.8.2
Invalid Schema File Name
10.2.8.3
Invalid Namespace in Schema File
10.2.8.4
Invalid Table Name in Schema File
10.3
Java Database Connectivity Metadata Provider
10.3.1
JDBC Detailed Functionality
10.3.2
Java Classpath
10.3.3
JDBC Metadata Provider Configuration
10.3.4
Sample Configuration
10.4
Hive Metadata Provider
10.4.1
Detailed Functionality
10.4.2
Configuring Hive with a Remote Metastore Database
10.4.3
Classpath Configuration
10.4.4
Hive Metadata Provider Configuration
10.4.5
Sample Configuration
10.4.6
Security
10.4.7
Metadata Change Event
10.4.8
Limitations
10.4.9
Additional Considerations
10.4.10
Troubleshooting
A
HDFS Handler Client Dependencies
A.1
Hadoop Client Dependencies
A.1.1
HDFS 2.7.1
A.1.2
HDFS 2.6.0
A.1.3
HDFS 2.5.2
A.1.4
HDFS 2.4.1
A.1.5
HDFS 2.3.0
A.1.6
HDFS 2.2.0
B
HBase Handler Client Dependencies
B.1
HBase Client Dependencies
B.1.1
HBase 1.1.1
B.1.2
HBase 1.0.1.1
C
Flume Handler Client Dependencies
C.1
Flume Client Dependencies
C.1.1
Flume 1.6.0
C.1.2
Flume 1.5.2
C.1.3
Flume 1.4.0
D
Kafka Handler Client Dependencies
D.1
Kafka Client Dependencies
D.1.1
Kafka 0.9.0.1
D.1.2
Kafka 0.10.0.1
E
Cassandra Handler Client Dependencies
E.1
Cassandra Datastax Java Driver 3.1.0
F
MongoDB Handler Client Dependencies
F.1
MongoDB Java Driver 3.2.2
Scripting on this page enhances content navigation, but does not change the content in any way.