1/15
Contents
Title and Copyright Information
Preface
Audience
Documentation Accessibility
Related Documents
Conventions
1
Overview
1.1
Overview
1.2
Java Environment Setup
1.3
Properties Files
1.4
Transaction Grouping
1.5
Running with Replicat
1.5.1
Replicat Configuration
1.5.2
Adding the Replicat Process
1.5.3
Replicat Grouping
1.5.4
Replicat Checkpointing
1.5.5
Unsupported Replicat Features
1.5.6
Mapping Functionality
1.6
Running with Extract
1.6.1
Extract Configuration
1.6.2
Adding the Extract Process
1.6.3
Extract Grouping
1.7
Logging
1.7.1
Extract or Replicat Process Logging
1.7.2
Java Layer Logging
1.8
Metadata Change Events
1.9
Configuration Property CDATA[] Wrapping
1.10
Using Regular Expression Search and Replace
1.10.1
Using Schema Data Replace
1.10.2
Using Content Data Replace
1.11
Using Identities in Oracle GoldenGate Credential Store
1.11.1
Creating a Credential Store
1.11.2
Adding Users to a Credential Store
1.11.3
Configuring Properties to Access the Credential Store
2
Using the HDFS Handler
2.1
Overview
2.2
Hive Handler Support
2.3
Writing into HDFS in Sequence File Format
2.3.1
Integrating with Hive
2.3.2
Understanding the Data Format
2.4
Runtime Prerequisites
2.4.1
Classpath Configuration
2.4.2
Pluggable Formatters
2.4.3
HDFS Handler Configuration
2.4.4
Sample Configuration
2.4.5
Troubleshooting the HDFS Handler
2.4.5.1
Java Classpath
2.4.5.2
HDFS Connection Properties
2.4.5.3
Handler and Formatter Configuration
2.4.6
Performance Considerations
2.4.7
Security
2.5
Writing in HDFS in Avro Object Container File Format
2.6
HDFS Handler Certification Matrix
2.7
Metadata Change Events
2.8
Partitioning
2.9
Common Pitfalls
2.10
Best Practices
3
Using the HBase Handler
3.1
Overview
3.2
HBase Handler Certification Matrix
3.3
Detailed Functionality
3.4
Runtime Prerequisites
3.4.1
Classpath Configuration
3.4.2
Pluggable Formatters
3.4.3
HBase Handler Configuration
3.4.4
Sample Configuration
3.4.5
Troubleshooting the HBase Handler
3.4.5.1
Java Classpath
3.4.5.2
HBase Connection Properties
3.4.5.3
Logging of Handler Configuration
3.4.6
Performance Considerations
3.4.7
Security
3.5
Metadata Change Events
3.6
Common Pitfalls
4
Using the Flume Handler
4.1
Overview
4.2
Runtime Prerequisites
4.3
Classpath Configuration
4.4
Pluggable Formatters
4.5
Flume Handler Configuration
4.6
Sample Configuration
4.7
Troubleshooting
4.7.1
Java Classpath
4.7.2
Flume Flow Control Issues
4.7.3
Flume Agent Configuration File Not Found
4.7.4
Flume Connection Exception
4.7.5
Other Failures
4.8
Data Mapping of Operations to Flume Events
4.8.1
Operation Mode
4.8.2
Transaction Mode and EventMapsTo Operation
4.8.3
Transaction Mode and EventMapsTo Transaction
4.9
Flume Handler Certification Matrix
4.10
Performance Considerations
4.11
Metadata Change Events
4.12
Example Flume Source Configuration
4.12.1
Avro Flume Source
4.12.2
Thrift Flume Source
4.13
Advanced Features
4.13.1
Schema Propagation
4.13.2
Security
4.13.3
Fail Over Functionality
4.13.4
Load Balancing Functionality
5
Using the Kafka Handler
5.1
Setup and Running
5.1.1
Runtime Prerequisites
5.1.2
Classpath Configuration
5.1.3
Pluggable Formatters
5.1.4
Kafka Handler Configuration
5.1.5
Sample Configuration
5.1.5.1
Java Adapter Properties File
5.1.6
Kafka Producer Configuration File
5.2
Detailed Functionality
5.2.1
Transaction versus Operation Mode
5.2.2
Blocking versus Blocking Mode
5.2.3
Publishing to Multiple Topics
5.3
Schema Propagation
5.4
Troubleshooting
5.4.1
Verify Kafka Setup
5.4.2
Classpath Issues
5.4.3
Invalid Kafka Version
5.4.4
Kafka Producer Properties File Not Found
5.4.5
Kafka Connection Problem
5.5
Performance Considerations
5.6
Security
5.7
Kafka Handler Certification Matrix
5.8
Metadata Change Events
5.9
Snappy Considerations
6
Using the Pluggable Formatters
6.1
Operation versus Row Based Formatting
6.1.1
Operation Formatters
6.1.2
Row Formatters
6.2
Delimited Text Formatter
6.2.1
Message Formatting Details
6.2.2
Sample Formatted Messages
6.2.2.1
Sample Insert Message
6.2.2.2
Sample Update Message
6.2.2.3
Sample Delete Message
6.2.2.4
Sample Truncate Message
6.2.3
Common Pitfalls
6.2.4
Logging of Output Format Summary
6.2.5
Delimited Text Format Configuration
6.2.6
Sample Configuration
6.2.7
Metadata Change Events
6.2.8
Special Considerations
6.2.8.1
Primary Key Updates
6.2.8.2
Data Consolidation
6.3
JSON Formatter
6.3.1
Message Formatting Details
6.3.1.1
Operation Metadata
6.3.1.2
Operation Data
6.3.2
Sample JSON Messages
6.3.2.1
Sample Insert Message
6.3.2.2
Sample Update Message
6.3.2.3
Sample Delete Message
6.3.2.4
Sample Truncate Message
6.3.3
JSON Schemas
6.3.4
JSON Schema Configuration
6.3.5
Sample Configuration
6.3.6
Metadata Change Events
6.3.7
Primary Key Updates
6.4
Avro Row Formatter
6.4.1
Message Formatting Details
6.4.1.1
Operation Metadata
6.4.1.2
Operation Data
6.4.2
Sample Avro Row Messages
6.4.2.1
Sample Insert Message
6.4.2.2
Sample Update Message
6.4.2.3
Sample Delete Message
6.4.2.4
Sample Truncate Message
6.4.3
Avro Schemas
6.4.4
Avro Row Configuration
6.4.5
Sample Configuration
6.4.6
Metadata Change Events
6.4.7
Special Considerations
6.4.7.1
Troubleshooting
6.4.7.2
Primary Key Updates
6.4.7.3
Generic Wrapper Functionality
6.5
Avro Operation Formatter
6.5.1
Message Formatting Details
6.5.1.1
Operation Metadata
6.5.1.2
Operation Data
6.5.2
Sample Avro Operation Messages
6.5.2.1
Sample Insert Message
6.5.2.2
Sample Update Message
6.5.2.3
Sample Delete Message
6.5.2.4
Sample Truncate Message
6.5.3
Avro Schema
6.5.4
Avro Operation Formatter Configuration
6.5.5
Sample Configuration
6.5.6
Metadata Change Events
6.5.7
Special Considerations
6.5.7.1
Troubleshooting
6.5.7.2
Primary Key Updates
6.5.7.3
Generic Wrapper Message
6.6
Avro Object Container File Formatter
6.6.1
Avro OCF Formatter Configuration
6.7
XML Formatter
6.7.1
Message Formatting Details
6.7.2
Sample XML Messages
6.7.2.1
Sample Insert Message
6.7.2.2
Sample Update Message
6.7.2.3
Sample Delete Message
6.7.2.4
Sample Truncate Message
6.7.3
XML Schema
6.7.4
XML Configuration
6.7.5
Sample Configuration
6.7.6
Metadata Change Events
6.7.7
Primary Key Updates
7
Using the Metadata Provider
7.1
Avro Metadata Provider
7.1.1
Detailed Functionality
7.1.2
Runtime Prerequisites
7.1.3
Classpath Configuration
7.1.4
Avro Metadata Provider Configuration
7.1.5
Sample Configuration
7.1.6
Metadata Change Event
7.1.7
Limitations
7.1.8
Troubleshooting
7.1.8.1
Invalid Schema Files Location
7.1.8.2
Invalid Schema File Name
7.1.8.3
Invalid Namespace in Schema File
7.1.8.4
Invalid Table Name in Schema File
7.2
Hive Metadata Provider
7.2.1
Detailed Functionality
7.2.2
Runtime Prerequisites
7.2.3
Classpath Configuration
7.2.4
Hive Metadata Provider Configuration
7.2.5
Sample Configuration
7.2.6
Security
7.2.7
Metadata Change Event
7.2.8
Limitations
7.2.9
Common Pitfalls
7.2.10
Troubleshooting
A
HBase Handler Client Dependencies
A.1
HBase Client Dependencies
A.1.1
HBase 1.1.1
A.1.2
HBase 1.0.1.1
B
HDFS Handler Client Dependencies
B.1
Hadoop Client Dependencies
B.1.1
HDFS 2.7.1
B.1.2
HDFS 2.6.0
B.1.3
HDFS 2.5.2
B.1.4
HDFS 2.4.1
B.1.5
HDFS 2.3.0
B.1.6
HDFS 2.2.0
C
Flume Handler Client Dependencies
C.1
Flume Client Dependencies
C.1.1
Flume 1.6.0
C.1.2
Flume 1.5.2
C.1.3
Flume 1.4.0
D
Kafka Handler Client Dependencies
D.1
Kafka Client Dependencies
D.1.1
Kafka 0.8.2.1
Scripting on this page enhances content navigation, but does not change the content in any way.