Contents

List of Tables

Title and Copyright Information

Preface

• Audience
• Documentation Accessibility
• Related Documents
• Conventions

1 Big Data Integration with Oracle Data Integrator

• 1.1 Overview of Hadoop Data Integration
• 1.2 Big Data Knowledge Modules Matrix

2 Hadoop Data Integration Concepts

• 2.1 Hadoop Data Integration with Oracle Data Integrator
• 2.2 Generate Code in Different Languages with Oracle Data Integrator
• 2.3 Leveraging Apache Oozie to execute Oracle Data Integrator Projects
• 2.4 Oozie Workflow Execution Modes

3 Setting Up the Environment for Integrating Hadoop Data

• 3.1 Configuring Big Data technologies using the Big Data Configurations Wizard
  • 3.1.1 General Settings
  • 3.1.2 HBase Data Server Definition
• 3.2 Creating and Initializing the Hadoop Data Server
  • 3.2.1 Hadoop Data Server Definition
  • 3.2.2 Hadoop Data Server Properties
• 3.3 Creating a Hadoop Physical Schema
• 3.4 Configuring the Oracle Data Integrator Agent to Execute Hadoop Jobs
• 3.5 Configuring Oracle Loader for Hadoop
• 3.6 Configuring Oracle Data Integrator to Connect to a Secure Cluster
• 3.7 Configuring Oracle Data Integrator Studio for Executing Hadoop Jobs on the Local Agent

4 Integrating Hadoop Data

• 4.1 Integrating Hadoop Data
• 4.2 Setting Up File Data Sources
• 4.3 Setting Up Hive Data Sources
• 4.4 Setting Up HBase Data Sources
• 4.5 Importing Hadoop Knowledge Modules
• 4.6 Creating a Oracle Data Integrator Model from a Reverse-Engineered Hive, HBase, and HDFS Models
  • 4.6.1 Creating a Model
  • 4.6.2 Reverse Engineering Hive Tables
  • 4.6.3 Reverse Engineering HBase Tables
  • 4.6.4 Reverse Engineering HDFS Files
• 4.7 Loading Data from Files into Hive
• 4.8 Loading Data from Hive to Files
• 4.9 Loading Data from HBase into Hive
• 4.10 Loading Data from Hive into Hbase
• 4.11 Loading Data from an SQL Database into Hive, HBase, and File using SQOOP
• 4.12 Loading Data from an SQL Database into Hive using SQOOP
• 4.13 Loading Data from an SQL Database into File using SQOOP
• 4.14 Loading Data from an SQL Database into HBase using SQOOP
• 4.15 Validating and Transforming Data Within Hive
• 4.16 Loading Data into an Oracle Database from Hive and File
• 4.17 Loading Data into an SQL Database from Hbase, Hive and File using SQOOP

5 Executing Oozie Workflows

• 5.1 Executing Oozie Workflows with Oracle Data Integrator
• 5.2 Setting Up and Initializing the Oozie Runtime Engine
  • 5.2.1 Oozie Runtime Engine Definition
  • 5.2.2 Oozie Runtime Engine Properties
• 5.3 Creating a Logical Oozie Engine
• 5.4 Executing or Deploying an Oozie Workflow
• 5.5 Auditing Hadoop Logs
• 5.6 Userlib jars support for running ODI Oozie workflows

6 Using Query Processing Engines to Generate Code in Different Languages

• 6.1 Query Processing Engines Supported by Oracle Data Integrator
• 6.2 Setting Up Hive Data Server
  • 6.2.1 Hive Data Server Definition
  • 6.2.2 Hive Data Server Connection Details
• 6.3 Creating a Hive Physical Schema
• 6.4 Setting Up Pig Data Server
  • 6.4.1 Pig Data Server Definition
  • 6.4.2 Pig Data Server Properties
• 6.5 Creating a Pig Physical Schema
• 6.6 Setting Up Spark Data Server
  • 6.6.1 Spark Data Server Definition
  • 6.6.2 Spark Data Server Properties
• 6.7 Creating a Spark Physical Schema
• 6.8 Generating Code in Different Languages

7 Working with Unstructured Data and Complex Data

• 7.1 Working with Unstructured Data
• 7.2 Working with Complex Data

A Hive Knowledge Modules

• A.1 LKM SQL to Hive SQOOP
• A.2 LKM SQL to File SQOOP Direct
• A.3 LKM SQL to HBase SQOOP Direct
• A.4 LKM File to SQL SQOOP
• A.5 LKM Hive to SQL SQOOP
• A.6 LKM HBase to SQL SQOOP
• A.7 IKM Hive Append
• A.8 LKM File to Hive LOAD DATA
• A.9 LKM File to Hive LOAD DATA Direct
• A.10 LKM HBase to Hive HBASE-SERDE
• A.11 LKM Hive to HBase Incremental Update HBASE-SERDE Direct
• A.12 LKM Hive to File Direct
• A.13 XKM Hive Sort
• A.14 LKM File to Oracle OLH-OSCH
• A.15 LKM File to Oracle OLH-OSCH Direct
• A.16 LKM Hive to Oracle OLH-OSCH
• A.17 LKM Hive to Oracle OLH-OSCH Direct
• A.18 RKM Hive
• A.19 RKM HBase
• A.20 IKM File to Hive (Deprecated)
• A.21 LKM HBase to Hive (HBase-SerDe) [Deprecated]
• A.22 IKM Hive to HBase Incremental Update (HBase-SerDe) [Deprecated]
• A.23 IKM SQL to Hive-HBase-File (SQOOP) [Deprecated]
• A.24 IKM Hive Control Append (Deprecated)
• A.25 CKM Hive (Deprecated)
• A.26 IKM Hive Transform (Deprecated)
• A.27 IKM File-Hive to Oracle (OLH-OSCH) [Deprecated]
• A.28 IKM File-Hive to SQL (SQOOP) [Deprecated]

B Pig Knowledge Modules

• B.1 LKM File to Pig
• B.2 LKM Pig to File
• B.3 LKM HBase to Pig
• B.4 LKM Pig to HBase
• B.5 LKM Hive to Pig
• B.6 LKM Pig to Hive
• B.7 LKM SQL to Pig SQOOP
• B.8 XKM Pig Aggregate
• B.9 XKM Pig Distinct
• B.10 XKM Pig Expression
• B.11 XKM Pig Filter
• B.12 XKM Pig Flatten
• B.13 XKM Pig Join
• B.14 XKM Pig Lookup
• B.15 XKM Pig Pivot
• B.16 XKM Pig Set
• B.17 XKM Pig Sort
• B.18 XKM Pig Split
• B.19 XKM Pig Subquery Filter
• B.20 XKM Pig Table Function
• B.21 XKM Pig Unpivot

C Spark Knowledge Modules

• C.1 LKM File to Spark
• C.2 LKM Spark to File
• C.3 LKM Hive to Spark
• C.4 LKM Spark to Hive
• C.5 XKM Spark Aggregate
• C.6 XKM Spark Distinct
• C.7 XKM Spark Expression
• C.8 XKM Spark Filter
• C.9 XKM Spark Flatten
• C.10 XKM Spark Join
• C.11 XKM Spark Lookup
• C.12 XKM Spark Pivot
• C.13 XKM Spark Set
• C.14 XKM Spark Sort
• C.15 XKM Spark Split
• C.16 XKM Spark Table Function
• C.17 IKM Spark Table Function
• C.18 XKM Spark Unpivot

D Components Knowledge Modules

• D.1 XKM Oracle Flatten
• D.2 XKM Oracle Flatten XML
• D.3 XKM Jagged

E Considerations, Limitations, and Issues

• E.1 Considerations, Limitations, and Issues

Index