Table of Contents
- Title and Copyright Information
- Preface
-
1
Introducing Oracle GoldenGate for Big Data
- 1.1 What’s Supported in Oracle GoldenGate for Big Data?
- 1.2 Setting Up Oracle GoldenGate for Big Data
-
1.3
Configuring Oracle GoldenGate for Big Data
- 1.3.1 Running with Replicat
- 1.3.2 Overview of Logging
- 1.3.3 About Schema Evolution and Metadata Change Events
- 1.3.4 About Configuration Property CDATA[] Wrapping
- 1.3.5 Using Regular Expression Search and Replace
- 1.3.6 Scaling Oracle GoldenGate for Big Data Delivery
- 1.3.7 Configuring Cluster High Availability
- 1.3.8 Using Identities in Oracle GoldenGate Credential Store
- 2 Using the BigQuery Handler
-
3
Using the Cassandra Handler
- 3.1 Overview
- 3.2 Detailing the Functionality
- 3.3 Setting Up and Running the Cassandra Handler
- 3.4 About Automated DDL Handling
- 3.5 Performance Considerations
- 3.6 Additional Considerations
- 3.7 Troubleshooting
- 4 Using the Elasticsearch Handler
- 5 Using the File Writer Handler
- 6 Using the HDFS Event Handler
- 7 Using the Optimized Row Columnar Event Handler
- 8 Configuring the ORC Event Handler
- 9 Using the Oracle Cloud Infrastructure Event Handler
- 10 Using the Parquet Event Handler
- 11 Using the S3 Event Handler
- 12 Using the Command Event Handler
- 13 Using the Redshift Event Handler
- 14 Using the Autonomous Data Warehouse Event Handler
- 15 Using the HBase Handler
-
16
Using the HDFS Handler
- 16.1 Overview
- 16.2 Writing into HDFS in SequenceFile Format
- 16.3 Setting Up and Running the HDFS Handler
- 16.4 Writing in HDFS in Avro Object Container File Format
- 16.5 Generating HDFS File Names Using Template Strings
- 16.6 Metadata Change Events
- 16.7 Partitioning
- 16.8 HDFS Additional Considerations
- 16.9 Best Practices
- 16.10 Troubleshooting the HDFS Handler
- 17 Using the Java Database Connectivity Handler
- 18 Using the Java Message Service Handler
-
19
Using the Kafka Handler
- 19.1 Overview
- 19.2 Detailed Functionality
- 19.3 Setting Up and Running the Kafka Handler
- 19.4 Schema Propagation
- 19.5 Performance Considerations
- 19.6 About Security
- 19.7 Metadata Change Events
- 19.8 Snappy Considerations
- 19.9 Kafka Interceptor Support
- 19.10 Kafka Partition Selection
- 19.11 Troubleshooting
- 20 Using the Kafka Connect Handler
-
21
Using the Kafka REST Proxy Handler
- 21.1 Overview
-
21.2
Setting Up and Starting the Kafka REST Proxy Handler Services
- 21.2.1 Using the Kafka REST Proxy Handler
- 21.2.2 Downloading the Dependencies
- 21.2.3 Classpath Configuration
- 21.2.4 Kafka REST Proxy Handler Configuration
- 21.2.5 Review a Sample Configuration
- 21.2.6 Security
- 21.2.7 Generating a Keystore or Truststore
- 21.2.8 Using Templates to Resolve the Topic Name and Message Key
- 21.2.9 Kafka REST Proxy Handler Formatter Properties
- 21.3 Consuming the Records
- 21.4 Performance Considerations
- 21.5 Kafka REST Proxy Handler Metacolumns Template Property
-
22
Using the Kinesis Streams Handler
- 22.1 Overview
- 22.2 Detailed Functionality
-
22.3
Setting Up and Running the Kinesis Streams Handler
- 22.3.1 Set the Classpath in Kinesis Streams Handler
- 22.3.2 Kinesis Streams Handler Configuration
- 22.3.3 Using Templates to Resolve the Stream Name and Partition Name
- 22.3.4 Configuring the Client ID and Secret in Kinesis Handler
- 22.3.5 Configuring the Proxy Server for Kinesis Streams Handler
- 22.3.6 Configuring Security in Kinesis Streams Handler
- 22.4 Kinesis Handler Performance Considerations
- 22.5 Troubleshooting
- 23 Using the MongoDB Handler
-
24
Using the Metadata Providers
- 24.1 About the Metadata Providers
- 24.2 Avro Metadata Provider
- 24.3 Java Database Connectivity Metadata Provider
-
24.4
Hive Metadata Provider
- 24.4.1 Detailed Functionality
- 24.4.2 Configuring Hive with a Remote Metastore Database
- 24.4.3 Classpath Configuration
- 24.4.4 Hive Metadata Provider Configuration Properties
- 24.4.5 Review a Sample Configuration
- 24.4.6 Security
- 24.4.7 Metadata Change Event
- 24.4.8 Limitations
- 24.4.9 Additional Considerations
- 24.4.10 Troubleshooting
- 25 Using the Oracle NoSQL Handler
-
26
Using the Pluggable Formatters
-
26.1
Using the Avro Formatter
- 26.1.1 Avro Row Formatter
-
26.1.2
The Avro Operation Formatter
- 26.1.2.1 Operation Metadata Formatting Details
- 26.1.2.2 Operation Data Formatting Details
- 26.1.2.3 Sample Avro Operation Messages
- 26.1.2.4 Avro Schema
- 26.1.2.5 Avro Operation Formatter Configuration Properties
- 26.1.2.6 Review a Sample Configuration
- 26.1.2.7 Metadata Change Events
- 26.1.2.8 Special Considerations
- 26.1.3 Avro Object Container File Formatter
- 26.1.4 Setting Metacolumn Output
-
26.2
Using the Delimited Text Formatter
-
26.2.1
Using the Delimited Text Row Formatter
- 26.2.1.1 Message Formatting Details
- 26.2.1.2 Sample Formatted Messages
- 26.2.1.3 Output Format Summary Log
- 26.2.1.4 Delimited Text Formatter Configuration Properties
- 26.2.1.5 Review a Sample Configuration
- 26.2.1.6 Metadata Change Events
- 26.2.1.7 Setting Metacolumn Output
- 26.2.1.8 Additional Considerations
- 26.2.2 Delimited Text Operation Formatter
-
26.2.1
Using the Delimited Text Row Formatter
-
26.3
Using the JSON Formatter
- 26.3.1 Operation Metadata Formatting Details
- 26.3.2 Operation Data Formatting Details
- 26.3.3 Row Data Formatting Details
- 26.3.4 Sample JSON Messages
- 26.3.5 JSON Schemas
- 26.3.6 JSON Formatter Configuration Properties
- 26.3.7 Review a Sample Configuration
- 26.3.8 Metadata Change Events
- 26.3.9 Setting Metacolumn Output
- 26.3.10 JSON Primary Key Updates
- 26.3.11 Integrating Oracle Stream Analytics
- 26.4 Using the Length Delimited Value Formatter
- 26.5 Using Operation-Based versus Row-Based Formatting
- 26.6 Using the XML Formatter
-
26.1
Using the Avro Formatter
-
27
Using Oracle GoldenGate Capture for Cassandra
- 27.1 Overview
- 27.2 Setting Up Cassandra Change Data Capture
- 27.3 Deduplication
- 27.4 Topology Changes
- 27.5 Data Availability in the CDC Logs
- 27.6 Using Extract Initial Load
- 27.7 Using Change Data Capture Extract
- 27.8 Replicating to RDMBS Targets
- 27.9 Partition Update or Insert of Static Columns
- 27.10 Partition Delete
- 27.11 Security and Authentication
- 27.12 Cleanup of CDC Commit Log Files
- 27.13 Multiple Extract Support
- 27.14 CDC Configuration Reference
- 27.15 Troubleshooting
- 28 Connecting to Microsoft Azure Data Lake
- 29 Connecting to Microsoft Azure Data Lake Gen 2
- 30 Connecting to Microsoft Azure Event Hubs
- 31 Connecting to Oracle Streaming Service
-
32
Stage and Merge Data Warehouse
Replication
-
32.1
Steps for Stage and Merge
- 32.1.1 Stage
- 32.1.2 Merge
- 32.1.3 Configuration of Handlers
- 32.1.4 File Writer Handler
- 32.1.5 Operation Aggregation
- 32.1.6 Object Store Event handler
- 32.1.7 JDBC Metadata Provider
- 32.1.8 Command Event handler Merge Script
- 32.1.9 Stage and Merge Sample Configuration
- 32.1.10 Variables in the Merge Script
- 32.1.11 SQL Statements in the Merge Script
- 32.1.12 Merge Script Functions
- 32.1.13 Prerequisites
- 32.1.14 Limitations
- 32.2 Snowflake Stage and Merge
- 32.3 Snowflake on AWS
- 32.4 Snowflake on Azure
- 32.5 Google BigQuery Stage and Merge
- 32.6 Hive Stage and Merge
-
32.1
Steps for Stage and Merge
- A Google BigQuery Dependancies
- B Cassandra Handler Client Dependencies
- C Cassandra Capture Client Dependencies
- D Elasticsearch Handler Transport Client Dependencies
- E Elasticsearch High Level REST Client Dependencies
- F HBase Handler Client Dependencies
- G HDFS Handler Client Dependencies
- H Kafka Handler Client Dependencies
- I Kafka Connect Handler Client Dependencies
- J MongoDB Handler Client Dependencies
- K Optimized Row Columnar Event Handler Client Dependencies
- L Parquet Event Handler Client Dependencies
- M Velocity Dependencies
- N OCI Dependencies
- O JMS Dependencies