1/24

Contents

Title and Copyright Information

Preface

• Audience
• Documentation Accessibility
• Related Information
• Conventions

1 Introduction to GoldenGate for Big Data

• 1.1 Introduction
• 1.2 Understanding What is Supported
  • 1.2.1 Verifying Certification and System Requirements
  • 1.2.2 Understanding Handler Compatibility
    • 1.2.2.1 Cassandra Handler
    • 1.2.2.2 Flume Handler
    • 1.2.2.3 Elasticsearch Handler
    • 1.2.2.4 HBase Handler
    • 1.2.2.5 HDFS Handler
    • 1.2.2.6 JBDC Handler
    • 1.2.2.7 Kafka and Kafka Connect Handlers
    • 1.2.2.8 Kinesis Streams Handler
    • 1.2.2.9 MongoDB Handler
  • 1.2.3 What are the Additional Support Considerations?
• 1.3 Setting Up Oracle GoldenGate for Big Data
  • 1.3.1 Java Environment Setup
  • 1.3.2 Properties Files
  • 1.3.3 Transaction Grouping
• 1.4 Configuring GoldenGate for Big Data
  • 1.4.1 Running with Replicat
    • 1.4.1.1 Configuring Replicat
    • 1.4.1.2 Adding the Replicat Process
    • 1.4.1.3 Replicat Grouping
    • 1.4.1.4 Replicat Checkpointing
    • 1.4.1.5 Initial Load Support
    • 1.4.1.6 Unsupported Replicat Features
    • 1.4.1.7 Mapping Functionality
  • 1.4.2 Logging
    • 1.4.2.1 Replicat Process Logging
    • 1.4.2.2 Java Layer Logging
  • 1.4.3 Schema Evolution and Metadata Change Events
  • 1.4.4 Configuration Property CDATA[] Wrapping
  • 1.4.5 Using Regular Expression Search and Replace
    • 1.4.5.1 Using Schema Data Replace
    • 1.4.5.2 Using Content Data Replace
  • 1.4.6 Scaling Oracle GoldenGate for Big Data Delivery
  • 1.4.7 Using Identities in Oracle GoldenGate Credential Store
    • 1.4.7.1 Creating a Credential Store
    • 1.4.7.2 Adding Users to a Credential Store
    • 1.4.7.3 Configuring Properties to Access the Credential Store

2 Using the Cassandra Handler

• 2.1 Overview
• 2.2 Detailed Functionality
  • 2.2.1 Cassandra Data Types
  • 2.2.2 Catalog, Schema, Table, and Column Name Mapping
  • 2.2.3 DDL Functionality
    • 2.2.3.1 Keyspaces
    • 2.2.3.2 Tables
    • 2.2.3.3 Add Column Functionality
    • 2.2.3.4 Drop Column Functionality
  • 2.2.4 Operation Processing
  • 2.2.5 Compressed Updates vs. Full Image Updates
  • 2.2.6 Primary Key Updates
• 2.3 Setting Up and Running the Cassandra Handler
  • 2.3.1 Cassandra Handler Configuration
  • 2.3.2 Sample Configuration
  • 2.3.3 Security
• 2.4 Automated DDL Handling
  • 2.4.1 Table Check and Reconciliation Process
• 2.5 Performance Considerations
• 2.6 Additional Considerations
• 2.7 Troubleshooting
  • 2.7.1 Java Classpath
  • 2.7.2 Logging
  • 2.7.3 Write Timeout Exception
  • 2.7.4 Logging

3 Using the Elasticsearch Handler

• 3.1 Overview
• 3.2 Detailed Functionality
  • 3.2.1 Elasticsearch Version
  • 3.2.2 Elasticsearch Index and Type
  • 3.2.3 Elasticsearch Document
  • 3.2.4 Elasticsearch Primary Key Update
  • 3.2.5 Elasticsearch Data Types
  • 3.2.6 Elasticsearch Operation Support
  • 3.2.7 Elasticsearch Connection
• 3.3 Setting Up and Running the Elasticsearch Handler
  • 3.3.1 Elasticsearch Handler Configuration
  • 3.3.2 Elasticsearch Transport Client Settings Properties File
• 3.4 Elasticsearch Performance Consideration
• 3.5 Elasticsearch Plug-in Support
• 3.6 Elasticsearch DDL Handling
• 3.7 Elasticsearch Operation Mode
• 3.8 Troubleshooting
  • 3.8.1 Incorrect Java Classpath
  • 3.8.2 Elasticsearch Version Mismatch
  • 3.8.3 Elasticsearch Transport Client Properties File Not Found
  • 3.8.4 Elasticsearch Cluster Connection Problem
  • 3.8.5 Elasticsearch Unsupported TRUNCATE Operation
  • 3.8.6 Elasticsearch Bulk Execute Errors
• 3.9 Logging
• 3.10 Known Issues in Elasticsearch Handler

4 Using the Flume Handler

• 4.1 Overview
• 4.2 Setting Up and Running the Flume Handler
  • 4.2.1 Classpath Configuration
  • 4.2.2 Flume Handler Configuration
  • 4.2.3 Sample Configuration
• 4.3 Data Mapping of Operations to Flume Events
  • 4.3.1 Operation Mode
  • 4.3.2 Transaction Mode and EventMapsTo Operation
  • 4.3.3 Transaction Mode and EventMapsTo Transaction
• 4.4 Performance Considerations
• 4.5 Metadata Change Events
• 4.6 Example Flume Source Configuration
  • 4.6.1 Avro Flume Source
  • 4.6.2 Thrift Flume Source
• 4.7 Advanced Features
  • 4.7.1 Schema Propagation
  • 4.7.2 Security
  • 4.7.3 Fail Over Functionality
  • 4.7.4 Load Balancing Functionality
• 4.8 Troubleshooting the Flume Handler
  • 4.8.1 Java Classpath
  • 4.8.2 Flume Flow Control Issues
  • 4.8.3 Flume Agent Configuration File Not Found
  • 4.8.4 Flume Connection Exception
  • 4.8.5 Other Failures

5 Using the HBase Handler

• 5.1 Overview
• 5.2 Detailed Functionality
• 5.3 Setting Up and Running the HBase Handler
  • 5.3.1 Classpath Configuration
  • 5.3.2 HBase Handler Configuration
  • 5.3.3 Sample Configuration
  • 5.3.4 Performance Considerations
  • 5.3.5 Security
• 5.4 Metadata Change Events
• 5.5 Additional Considerations
• 5.6 Troubleshooting the HBase Handler
  • 5.6.1 Java Classpath
  • 5.6.2 HBase Connection Properties
  • 5.6.3 Logging of Handler Configuration
  • 5.6.4 HBase Handler Delete-Insert Problem
  • 5.6.5 Cloudera CDH HBase Compatibility

6 Using the HDFS Handler

• 6.1 Overview
• 6.2 Writing into HDFS in SequenceFile Format
  • 6.2.1 Integrating with Hive
  • 6.2.2 Understanding the Data Format
  • 6.2.3 Setting Up and Running the HDFS Handler
    • 6.2.3.1 Classpath Configuration
    • 6.2.3.2 HDFS Handler Configuration
    • 6.2.3.3 Sample Configuration
    • 6.2.3.4 Performance Considerations
    • 6.2.3.5 Security
• 6.3 Writing in HDFS in Avro Object Container File Format
• 6.4 Generating HDFS File Names Using Template Strings
• 6.5 Metadata Change Events
• 6.6 Partitioning
• 6.7 Additional Considerations
• 6.8 Best Practices
• 6.9 Troubleshooting the HDFS Handler
  • 6.9.1 Java Classpath
  • 6.9.2 HDFS Connection Properties
  • 6.9.3 Handler and Formatter Configuration

7 Using the Java Database Connectivity Handler

• 7.1 Overview
• 7.2 Detailed Functionality
  • 7.2.1 Single Operation Mode
  • 7.2.2 Oracle Database Data Types
  • 7.2.3 MySQL Database Data Types
  • 7.2.4 Netezza Database Data Types
  • 7.2.5 Redshift Database Data Types
• 7.3 Setting Up and Running the JDBC Handler
  • 7.3.1 Java Classpath
  • 7.3.2 Handler Configuration
  • 7.3.3 Statement Caching
  • 7.3.4 Setting Up Error Handling
• 7.4 Sample Configurations
  • 7.4.1 Sample Oracle Database Target
  • 7.4.2 Sample Oracle Database Target with JDBC Metadata Provider
  • 7.4.3 Sample MySQL Database Target
  • 7.4.4 Sample MySQL Database Target with JDBC Metadata Provider

8 Using the Kafka Handler

• 8.1 Overview
• 8.2 Detailed Functionality
• 8.3 Setting Up and Running the Kafka Handler
  • 8.3.1 Classpath Configuration
  • 8.3.2 Kafka Handler Configuration
  • 8.3.3 Java Adapter Properties File
  • 8.3.4 Kafka Producer Configuration File
  • 8.3.5 Using Templates to Resolve the Topic Name and Message Key
• 8.4 Schema Propagation
• 8.5 Performance Considerations
• 8.6 Security
• 8.7 Metadata Change Events
• 8.8 Snappy Considerations
• 8.9 Troubleshooting
  • 8.9.1 Verify the Kafka Setup
  • 8.9.2 Classpath Issues
  • 8.9.3 Invalid Kafka Version
  • 8.9.4 Kafka Producer Properties File Not Found
  • 8.9.5 Kafka Connection Problem

9 Using the Kafka Connect Handler

• 9.1 Overview
• 9.2 Detailed Functionality
• 9.3 Setting Up and Running the Kafka Connect Handler
  • 9.3.1 Kafka Connect Handler Configuration
  • 9.3.2 Using Templates to Resolve the Topic Name and Message Key
  • 9.3.3 Configuring Security in Kafka Connect Handler
• 9.4 Kafka Connect Handler Performance Considerations
• 9.5 Troubleshooting the Kafka Connect Handler
  • 9.5.1 Java Classpath for Kafka Connect Handler
  • 9.5.2 Invalid Kafka Version
  • 9.5.3 Kafka Producer Properties File Not Found
  • 9.5.4 Kafka Connection Problem

10 Using the Kinesis Streams Handler

• 10.1 Overview
• 10.2 Detailed Functionality
  • 10.2.1 Amazon Kinesis Java SDK
  • 10.2.2 Kinesis Streams Input Limits
• 10.3 Setting Up and Running the Kinesis Streams Handler
  • 10.3.1 Set the Classpath in Kinesis Streams Handler
  • 10.3.2 Kinesis Streams Handler Configuration
  • 10.3.3 Using Templates to Resolve the Stream Name and Partition Name
  • 10.3.4 Configuring the Client ID and Secret in Kinesis Handler
  • 10.3.5 Configuring the Proxy Server for Kinesis Streams Handler
  • 10.3.6 Configuring Security in Kinesis Streams Handler
• 10.4 Kinesis Handler Performance Consideration
  • 10.4.1 Kinesis Streams Input Limitations
  • 10.4.2 Transaction Batching
  • 10.4.3 Deferring Flush at Transaction Commit
• 10.5 Troubleshooting
  • 10.5.1 Java Classpath
  • 10.5.2 Kinesis Handler Connectivity Issues
  • 10.5.3 Logging

11 Using the MongoDB Handler

• 11.1 Overview
• 11.2 Detailed Functionality
  • 11.2.1 Document Key Column
  • 11.2.2 Primary Key Update Operation
  • 11.2.3 MongoDB Trail Data Types
• 11.3 Setting Up and Running the MongoDB Handler
  • 11.3.1 Classpath Configuration
  • 11.3.2 MongoDB Handler Configuration
  • 11.3.3 Connecting and Authenticating
  • 11.3.4 Using Bulk Write
  • 11.3.5 Using Write Concern
  • 11.3.6 Using Three-Part Table Names
  • 11.3.7 Using Undo Handling
• 11.4 Sample Configuration

12 Using the Metadata Provider

• 12.1 About the Metadata Provider
• 12.2 Avro Metadata Provider
  • 12.2.1 Detailed Functionality
  • 12.2.2 Runtime Prerequisites
  • 12.2.3 Classpath Configuration
  • 12.2.4 Avro Metadata Provider Configuration
  • 12.2.5 Sample Configuration
  • 12.2.6 Metadata Change Event
  • 12.2.7 Limitations
  • 12.2.8 Troubleshooting
    • 12.2.8.1 Invalid Schema Files Location
    • 12.2.8.2 Invalid Schema File Name
    • 12.2.8.3 Invalid Namespace in Schema File
    • 12.2.8.4 Invalid Table Name in Schema File
• 12.3 Java Database Connectivity Metadata Provider
  • 12.3.1 JDBC Detailed Functionality
  • 12.3.2 Java Classpath
  • 12.3.3 JDBC Metadata Provider Configuration
  • 12.3.4 Sample Configuration
• 12.4 Hive Metadata Provider
  • 12.4.1 Detailed Functionality
  • 12.4.2 Configuring Hive with a Remote Metastore Database
  • 12.4.3 Classpath Configuration
  • 12.4.4 Hive Metadata Provider Configuration
  • 12.4.5 Sample Configuration
  • 12.4.6 Security
  • 12.4.7 Metadata Change Event
  • 12.4.8 Limitations
  • 12.4.9 Additional Considerations
  • 12.4.10 Troubleshooting

13 Using the Pluggable Formatters

• 13.1 Operation versus Row Based Formatting
  • 13.1.1 Operation Formatters
  • 13.1.2 Row Formatters
  • 13.1.3 Table Row or Column Value States
• 13.2 Delimited Text Formatter
  • 13.2.1 Message Formatting Details
  • 13.2.2 Sample Formatted Messages
    • 13.2.2.1 Sample Insert Message
    • 13.2.2.2 Sample Update Message
    • 13.2.2.3 Sample Delete Message
    • 13.2.2.4 Sample Truncate Message
  • 13.2.3 Additional Considerations
    • 13.2.3.1 Primary Key Updates
    • 13.2.3.2 Data Consolidation
  • 13.2.4 Output Format Summary Log
  • 13.2.5 Delimited Text Format Configuration
  • 13.2.6 Sample Configuration
  • 13.2.7 Metadata Change Events
• 13.3 JSON Formatter
  • 13.3.1 Operation Metadata Formatting Details
  • 13.3.2 Operation Data Formatting Details
  • 13.3.3 Row Data Formatting Details
  • 13.3.4 Sample JSON Messages
    • 13.3.4.1 Sample Operation Modeled JSON Messages
    • 13.3.4.2 Sample Flattened Operation Modeled JSON Messages
    • 13.3.4.3 Sample Row Modeled JSON Messages
    • 13.3.4.4 Sample Primary Key Output JSON Message
  • 13.3.5 JSON Schemas
  • 13.3.6 JSON Formatter Configuration
  • 13.3.7 Sample Configuration
  • 13.3.8 Metadata Change Events
  • 13.3.9 JSON Primary Key Updates
  • 13.3.10 Integrating Oracle Stream Analytics
• 13.4 Avro Formatter
  • 13.4.1 Avro Row Formatter
    • 13.4.1.1 Operation Metadata Formatting Details
    • 13.4.1.2 Operation Data Formatting Details
    • 13.4.1.3 Sample Avro Row Messages
      • 13.4.1.3.1 Sample Insert Message
      • 13.4.1.3.2 Sample Update Message
      • 13.4.1.3.3 Sample Delete Message
      • 13.4.1.3.4 Sample Truncate Message
    • 13.4.1.4 Avro Schemas
    • 13.4.1.5 Avro Row Configuration
    • 13.4.1.6 Sample Configuration
    • 13.4.1.7 Metadata Change Events
    • 13.4.1.8 Special Considerations
      • 13.4.1.8.1 Troubleshooting
      • 13.4.1.8.2 Primary Key Updates
      • 13.4.1.8.3 Generic Wrapper Functionality
  • 13.4.2 Avro Operation Formatter
    • 13.4.2.1 Operation Metadata Formatting Details
    • 13.4.2.2 Operation Data Formatting Details
    • 13.4.2.3 Sample Avro Operation Messages
      • 13.4.2.3.1 Sample Insert Message
      • 13.4.2.3.2 Sample Update Message
      • 13.4.2.3.3 Sample Delete Message
      • 13.4.2.3.4 Sample Truncate Message
    • 13.4.2.4 Avro Schema
    • 13.4.2.5 Avro Operation Formatter Configuration
    • 13.4.2.6 Sample Configuration
    • 13.4.2.7 Metadata Change Events
    • 13.4.2.8 Special Considerations
      • 13.4.2.8.1 Troubleshooting
      • 13.4.2.8.2 Primary Key Updates
      • 13.4.2.8.3 Generic Wrapper Message
  • 13.4.3 Avro Object Container File Formatter
    • 13.4.3.1 Avro OCF Formatter Configuration
• 13.5 XML Formatter
  • 13.5.1 Message Formatting Details
  • 13.5.2 Sample XML Messages
    • 13.5.2.1 Sample Insert Message
    • 13.5.2.2 Sample Update Message
    • 13.5.2.3 Sample Delete Message
    • 13.5.2.4 Sample Truncate Message
  • 13.5.3 XML Schema
  • 13.5.4 XML Configuration
  • 13.5.5 Sample Configuration
  • 13.5.6 Metadata Change Events
  • 13.5.7 Primary Key Updates

A Cassandra Handler Client Dependencies

• A.1 Cassandra Datastax Java Driver 3.1.0

B Elasticsearch Handler Client Dependencies

• B.1 Elasticsearch Handler Client Dependencies
• B.2 Elasticsearch 2.4.4 and Shield Plugin 2.2.2
• B.3 Elasticsearch 5.1.2 with X-Pack 5.1.2

C Flume Handler Client Dependencies

• C.1 Flume Client Dependencies
  • C.1.1 Flume 1.7.0
  • C.1.2 Flume 1.6.0
  • C.1.3 Flume 1.5.2
  • C.1.4 Flume 1.4.0

D HBase Handler Client Dependencies

• D.1 HBase Client Dependencies
  • D.1.1 HBase 1.2.5
  • D.1.2 HBase 1.1.1
  • D.1.3 HBase 1.0.1.1

E HDFS Handler Client Dependencies

• E.1 Hadoop Client Dependencies
  • E.1.1 HDFS 2.8.0
  • E.1.2 HDFS 2.7.1
  • E.1.3 HDFS 2.6.0
  • E.1.4 HDFS 2.5.2
  • E.1.5 HDFS 2.4.1
  • E.1.6 HDFS 2.3.0
  • E.1.7 HDFS 2.2.0

F Kafka Handler Client Dependencies

• F.1 Kafka Client Dependencies
  • F.1.1 Kafka 0.11.0.0
  • F.1.2 Kafka 0.10.2.0
  • F.1.3 Kafka 0.10.1.1
  • F.1.4 Kafka 0.10.0.1
  • F.1.5 Kafka 0.9.0.1

G Kafka Connect Handler Client Dependencies

• G.1 Kafka Connect Client Dependencies
  • G.1.1 Kafka  0.11.0.0
  • G.1.2 Kafka 0.10.2.0
  • G.1.3 Kafka 0.10.2.0
  • G.1.4 Kafka 0.10.0.0
  • G.1.5 Kafka 0.9.0.1
• G.2 Confluent IO Avro Converter and Schema Registry
  • G.2.1 Confluent IO 3.2.1
  • G.2.2 Confluent IO 3.2.0
  • G.2.3 Confluent IO 3.2.1
  • G.2.4 Confluent IO 3.1.1
  • G.2.5 Confluent IO 3.0.1
  • G.2.6 Confluent IO 2.0.1
  • G.2.7 Confluent IO 2.0.1

H MongoDB Handler Client Dependencies

• H.1 MongoDB Java Driver 3.2.2