Go to main content
1/30
Contents
List of Examples
List of Figures
List of Tables
Title and Copyright Information
Preface
Audience
Documentation Accessibility
Related Documents
Conventions
1
Big Data Integration with Oracle Data Integrator
1.1
Overview of Hadoop Data Integration
1.2
Big Data Knowledge Modules Matrix
2
Hadoop Data Integration Concepts
2.1
Hadoop Data Integration with Oracle Data Integrator
2.2
Generate Code in Different Languages with Oracle Data Integrator
2.3
Leveraging Apache Oozie to execute Oracle Data Integrator Projects
2.4
Oozie Workflow Execution Modes
2.5
Lambda Architecture
2.6
Spark Checkpointing
2.7
Spark Windowing and Stateful Aggregation
2.8
Spark Repartitioning and Caching
2.9
Kafka Integration with Oracle Data Integrator
3
Setting Up the Environment for Integrating Hadoop Data
3.1
Configuring Big Data technologies using the Big Data Configurations Wizard
3.1.1
General Settings
3.1.2
HDFS Data Server Definition
3.1.3
HBase Data Server Definition
3.1.4
Kafka Data Server Definition
3.1.5
Kafka Data Server Properties
3.2
Creating and Initializing the Hadoop Data Server
3.2.1
Hadoop Data Server Definition
3.2.2
Hadoop Data Server Properties
3.3
Creating a Hadoop Physical Schema
3.4
Configuring the Oracle Data Integrator Agent to Execute Hadoop Jobs
3.5
Configuring Oracle Loader for Hadoop
3.6
Configuring Oracle Data Integrator to Connect to a Secure Cluster
3.7
Configuring Oracle Data Integrator Studio for Executing Hadoop Jobs on the Local Agent
4
Integrating Hadoop Data
4.1
Integrating Hadoop Data
4.2
Setting Up File Data Sources
4.3
Setting Up HDFS Data Sources
4.4
Setting Up Hive Data Sources
4.5
Setting Up HBase Data Sources
4.6
Setting Up Kafka Data Sources
4.7
Setting Up Cassandra Data Sources
4.8
Importing Hadoop Knowledge Modules
4.9
Creating a Oracle Data Integrator Model from a Reverse-Engineered Hive, HBase, and HDFS Models
4.9.1
Creating a Model
4.9.2
Reverse Engineering Hive Tables
4.9.3
Reverse Engineering HBase Tables
4.9.4
Reverse Engineering HDFS Tables
4.9.5
Reverse Engineering Cassandra Tables
4.10
Loading Data from Files into Hive
4.11
Loading Data from Hive to Files
4.12
Loading Data from HBase into Hive
4.13
Loading Data from Hive into Hbase
4.14
Loading Data from an SQL Database into Hive, HBase, and File using SQOOP
4.15
Loading Data from an SQL Database into Hive using SQOOP
4.16
Loading Data from an SQL Database into File using SQOOP
4.17
Loading Data from an SQL Database into HBase using SQOOP
4.18
Validating and Transforming Data Within Hive
4.19
Loading Data into an Oracle Database from Hive and File
4.20
Loading Data into an SQL Database from Hbase, Hive and File using SQOOP
4.21
Loading Data from Kafka to Spark
5
Executing Oozie Workflows
5.1
Executing Oozie Workflows with Oracle Data Integrator
5.2
Setting Up and Initializing the Oozie Runtime Engine
5.2.1
Oozie Runtime Engine Definition
5.2.2
Oozie Runtime Engine Properties
5.3
Creating a Logical Oozie Engine
5.4
Executing or Deploying an Oozie Workflow
5.5
Auditing Hadoop Logs
5.6
Userlib jars support for running ODI Oozie workflows
6
Spark Streaming Support
6.1
Enabling Streaming Support for Oracle Data Integrator
6.2
Enabling Streaming Support
6.2.1
Spark Streaming DataServer Properties
6.2.2
Extra Spark Streaming Data Properties
6.3
Execute Mapping in Streaming Mode
7
Using Query Processing Engines to Generate Code in Different Languages
7.1
Query Processing Engines Supported by Oracle Data Integrator
7.2
Setting Up Hive Data Server
7.2.1
Hive Data Server Definition
7.2.2
Hive Data Server Connection Details
7.3
Creating a Hive Physical Schema
7.4
Setting Up Pig Data Server
7.4.1
Pig Data Server Definition
7.4.2
Pig Data Server Properties
7.5
Creating a Pig Physical Schema
7.6
Setting Up Spark Data Server
7.6.1
Spark Data Server Definition
7.6.2
Spark Data Server Properties
7.7
Creating a Spark Physical Schema
7.8
Generating Code in Different Languages
8
Working with Unstructured Data
8.1
Working with Unstructured Data
9
Working with Complex files
9.1
HDFS Formats
9.2
Working with Complex Files
9.3
Identifying, Adding and Removing Flattened Attributes
9.4
Loading Data from HDFS File to Hive
9.5
Loading Data from HDFS File to Spark
A
Hive Knowledge Modules
A.1
LKM SQL to Hive SQOOP
A.2
LKM SQL to File SQOOP Direct
A.3
LKM SQL to HBase SQOOP Direct
A.4
LKM File to SQL SQOOP
A.5
LKM Hive to SQL SQOOP
A.6
LKM HBase to SQL SQOOP
A.7
LKM HDFS File to Hive
A.8
LKM HDFS File to Hive (Direct)
A.9
IKM Hive Append
A.10
IKM Hive Incremental Update
A.11
LKM File to Hive LOAD DATA
A.12
LKM File to Hive LOAD DATA Direct
A.13
LKM HBase to Hive HBASE-SERDE
A.14
LKM Hive to HBase Incremental Update HBASE-SERDE Direct
A.15
LKM Hive to File Direct
A.16
XKM Hive Sort
A.17
LKM File to Oracle OLH-OSCH
A.18
LKM File to Oracle OLH-OSCH Direct
A.19
LKM Hive to Oracle OLH-OSCH
A.20
LKM Hive to Oracle OLH-OSCH Direct
A.21
RKM Hive
A.22
RKM HBase
A.23
IKM File to Hive (Deprecated)
A.24
LKM HBase to Hive (HBase-SerDe) [Deprecated]
A.25
IKM Hive to HBase Incremental Update (HBase-SerDe) [Deprecated]
A.26
IKM SQL to Hive-HBase-File (SQOOP) [Deprecated]
A.27
IKM Hive Control Append (Deprecated)
A.28
CKM Hive (Deprecated)
A.29
IKM Hive Transform (Deprecated)
A.30
IKM File-Hive to Oracle (OLH-OSCH) [Deprecated]
A.31
IKM File-Hive to SQL (SQOOP) [Deprecated]
B
Pig Knowledge Modules
B.1
LKM File to Pig
B.2
LKM Pig to File
B.3
LKM HBase to Pig
B.4
LKM Pig to HBase
B.5
LKM Hive to Pig
B.6
LKM Pig to Hive
B.7
LKM SQL to Pig SQOOP
B.8
XKM Pig Aggregate
B.9
XKM Pig Distinct
B.10
XKM Pig Expression
B.11
XKM Pig Filter
B.12
XKM Pig Flatten
B.13
XKM Pig Join
B.14
XKM Pig Lookup
B.15
XKM Pig Pivot
B.16
XKM Pig Set
B.17
XKM Pig Sort
B.18
XKM Pig Split
B.19
XKM Pig Subquery Filter
B.20
XKM Pig Table Function
B.21
XKM Pig Unpivot
C
Spark Knowledge Modules
C.1
LKM File to Spark
C.2
LKM Spark to File
C.3
LKM Hive to Spark
C.4
LKM Spark to Hive
C.5
LKM HDFS to Spark
C.6
LKM Spark to HDFS
C.7
LKM Kafka to Spark
C.8
LKM Spark to Kafka
C.9
LKM SQL to Spark
C.10
LKM Spark to SQL
C.11
RKM Cassandra
C.12
XKM Spark Aggregate
C.13
XKM Spark Distinct
C.14
XKM Spark Expression
C.15
XKM Spark Filter
C.16
XKM Spark Input Signature and Output Signature
C.17
XKM Spark Join
C.18
XKM Spark Lookup
C.19
XKM Spark Pivot
C.20
XKM Spark Set
C.21
XKM Spark Sort
C.22
XKM Spark Split
C.23
XKM Spark Table Function
C.24
IKM Spark Table Function
C.25
XKM Spark Unpivot
D
Component Knowledge Modules
D.1
XKM Oracle Flatten
D.2
XKM Oracle Flatten XML
D.3
XKM Spark Flatten
D.4
XKM Jagged
E
Considerations, Limitations, and Issues
E.1
Considerations, Limitations, and Issues
Scripting on this page enhances content navigation, but does not change the content in any way.