Contents

Title and Copyright Information

Preface

• Audience
• Documentation Accessibility
• Related Documents
• Conventions

What's New in Oracle Big Data Connectors?

• Changes in Release 1.1

1 Getting Started with Oracle Big Data Connectors

• 1.1 About Oracle Big Data Connectors
• 1.2 Big Data Concepts and Technologies
  • 1.2.1 What is MapReduce?
  • 1.2.2 What is Apache Hadoop?
• 1.3 Downloading the Oracle Big Data Connectors Software
• 1.4 Oracle Direct Connector for Hadoop Distributed File System Setup
  • 1.4.1 Required Software
  • 1.4.2 Installing and Configuring Hadoop
  • 1.4.3 Installing Oracle Direct Connector for HDFS
  • 1.4.4 Granting User Access to Oracle Direct Connector for HDFS
• 1.5 Oracle Loader for Hadoop Setup
  • 1.5.1 Required Software
  • 1.5.2 Installing Oracle Loader for Hadoop
• 1.6 Oracle Data Integrator Application Adapter for Hadoop Setup
  • 1.6.1 System Requirements and Certifications
  • 1.6.2 Technology-Specific Requirements
  • 1.6.3 Location of ODI Application Adapter for Hadoop
  • 1.6.4 Setting Up the Topology
• 1.7 Oracle R Connector for Hadoop Setup
  • 1.7.1 Installing the Software on Hadoop
    • 1.7.1.1 Software Requirements for a Third-Party Hadoop Cluster
    • 1.7.1.2 Installing Sqoop on a Hadoop Cluster
    • 1.7.1.3 Installing R on a Hadoop Cluster
    • 1.7.1.4 Installing the ORCH Package on a Hadoop Cluster
  • 1.7.2 Providing Remote Client Access to R Users
    • 1.7.2.1 Software Requirements for Remote Client Access
    • 1.7.2.2 Configuring the Server as a Hadoop Client
    • 1.7.2.3 Installing Sqoop on a Hadoop Client
    • 1.7.2.4 Installing R on a Hadoop Client
    • 1.7.2.5 Installing the ORCH Package on a Hadoop Client
    • 1.7.2.6 Installing the Oracle R Enterprise Client Packages (Optional)

2 Oracle Direct Connector for Hadoop Distributed File System

• 2.1 About Oracle Direct Connector for HDFS
• 2.2 Creating an External Table for HDFS
  • 2.2.1 Basic SQL Syntax for the External Table
  • 2.2.2 Testing the External Table
  • 2.2.3 External Table Example
• 2.3 Publishing the HDFS Data Paths
  • 2.3.1 ExternalTable Command
    • 2.3.1.1 Altering HADOOP_CLASSPATH
    • 2.3.1.2 ExternalTable Command Syntax
    • 2.3.1.3 ExternalTable Command Example
  • 2.3.2 How to Publish Data Pump Files
  • 2.3.3 Creating a Configuration File
  • 2.3.4 Configuration Properties
    • 2.3.4.1 Property Descriptions
• 2.4 Querying Data in HDFS

3 Oracle Loader for Hadoop

• 3.1 What Is Oracle Loader for Hadoop?
• 3.2 Using Oracle Loader for Hadoop
  • 3.2.1 Implementing InputFormat
    • 3.2.1.1 HiveToAvroInputFormat
    • 3.2.1.2 DelimitedTextInputFormat
  • 3.2.2 Creating the loaderMap Document
    • 3.2.2.1 Example loaderMap Document
  • 3.2.3 Accessing Table Metadata
    • 3.2.3.1 Running the OraLoaderMetadata Utility
  • 3.2.4 Invoking OraLoader
  • 3.2.5 Loading Files Into an Oracle Database (Offline Loads Only)
    • 3.2.5.1 Loading From Delimited Text Files Into an Oracle Database
• 3.3 Output Modes During OraLoader Invocation
  • 3.3.1 JDBC Output
  • 3.3.2 Oracle OCI Direct Path Output
  • 3.3.3 Delimited Text Output
  • 3.3.4 Oracle Data Pump Output
• 3.4 Balancing Loads When Loading Data into Partitioned Tables
  • 3.4.1 Using the Sampling Feature
  • 3.4.2 Tuning Load Balancing and Sampling Behavior
    • 3.4.2.1 Properties to Tune Load Balancing
    • 3.4.2.2 Properties to Tune Sampling Behavior
  • 3.4.3 Does Oracle Loader for Hadoop Always Use the Sampler's Partitioning Scheme?
  • 3.4.4 What Happens When a Sampling Feature Property Has an Invalid Value?
• 3.5 Primary Configuration Properties for the Load Balancing Feature
• 3.6 OraLoader Configuration Properties
• 3.7 Example of Using Oracle Loader for Hadoop
• 3.8 Target Table Characteristics
  • 3.8.1 Supported Data Types
  • 3.8.2 Supported Partitioning Strategies
• 3.9 Loader Map XML Schema Definition
• 3.10 OraLoader for Hadoop Configuration Properties
• 3.11 Third-Party Licenses for Bundled Software
  • 3.11.1 Apache Licensed Code
  • 3.11.2 Apache Avro avro-1.5.4.jar
  • 3.11.3 Apache Commons Mathematics Library 2.2
  • 3.11.4 Jackon JSON Library 1.5.2

4 Oracle Data Integrator Application Adapter for Hadoop

• 4.1 Introduction
  • 4.1.1 Concepts
  • 4.1.2 Knowledge Modules
  • 4.1.3 Security
• 4.2 Setting Up the Topology
  • 4.2.1 Setting Up File Data Sources
  • 4.2.2 Setting Up Hive Data Sources
  • 4.2.3 Setting Up the Oracle Data Integrator Agent to Execute Hadoop Jobs
  • 4.2.4 Configuring Oracle Data Integrator Studio for Executing Hadoop Jobs on the Local Agent
• 4.3 Setting Up an Integration Project
• 4.4 Creating an Oracle Data Integrator Model from a Reverse-Engineered Hive Model
  • 4.4.1 Creating a Model
  • 4.4.2 Reverse Engineering Hive Tables
• 4.5 Designing the Interface
  • 4.5.1 Loading Data from Files into Hive
  • 4.5.2 Validating and Transforming Data Within Hive
    • 4.5.2.1 IKM Hive Control Append
    • 4.5.2.2 CKM Hive
    • 4.5.2.3 IKM Hive Transform
  • 4.5.3 Loading Data into an Oracle Database from Hive and HDFS

5 Oracle R Connector for Hadoop

• 5.1 About Oracle R Connector for Hadoop
  • 5.1.1 Oracle R Connector for Hadoop APIs
  • 5.1.2 Access to Oracle Database
• 5.2 Scenarios for Using Oracle R Packages
• 5.3 Security Notes for Oracle R Connector for Hadoop
• 5.4 Functions in Alphabetical Order
• 5.5 Functions by Category
  • 5.5.1 Making Connections
  • 5.5.2 Copying Data
  • 5.5.3 Exploring Files
  • 5.5.4 Writing MapReduce Functions
  • 5.5.5 Debugging Scripts
  • 5.5.6 Executing Scripts
• 5.6 ORCH mapred.config Class
• 5.7 Example R Programs Using Oracle R Connector for Hadoop
  • 5.7.1 Using the Examples
  • 5.7.2 Using the Demos
• hadoop.exec
• hadoop.run
• hdfs.attach
• hdfs.cd
• hdfs.cp
• hdfs.describe
• hdfs.download
• hdfs.exists
• hdfs.get
• hdfs.id
• hdfs.ls
• hdfs.mkdir
• hdfs.mv
• hdfs.parts
• hdfs.pull
• hdfs.push
• hdfs.put
• hdfs.pwd
• hdfs.rm
• hdfs.rmdir
• hdfs.root
• hdfs.sample
• hdfs.setroot
• hdfs.size
• hdfs.upload
• is.hdfs.id
• orch.connect
• orch.dbcon
• orch.dbg.off
• orch.dbg.on
• orch.dbg.output
• orch.dbinfo
• orch.disconnect
• orch.dryrun
• orch.export
• orch.keyval
• orch.keyvals
• orch.pack
• orch.reconnect
• orch.unpack
• orch.version

Index