Table of Contents

• Title and Copyright Information
• Preface
  • Audience
  • Documentation Accessibility
  • Related Documents
  • Text Conventions
  • Syntax Conventions
  • Changes in Oracle Big Data Connectors 5.1
• Part I Setup
  • 1 Getting Started with Oracle Big Data Connectors
    • About Oracle Big Data Connectors
    • Big Data Concepts and Technologies
      • What is MapReduce?
      • What is Apache Hadoop?
    • Download and Install Oracle Big Data Connectors
    • Certified Hadoop Platforms
    • Secure Connection to Oracle Database
      • Using JDBC SSL
      • Using Secure External Java KeyStore and Hadoop credential command
    • Oracle SQL Connector for Hadoop Distributed File System Setup
      • Software Requirements
      • Install and Configure a Hadoop Client on the Oracle Database System
      • Install and Configure Oracle SQL Connector for HDFS
      • Oracle Database Privileges for OSCH Users
      • OS-Level Requirements for OSCH Users
      • Use Oracle SQL Connector for HDFS on a Secure Hadoop Cluster
    • Oracle Loader for Hadoop Setup
      • Software Requirements
      • Install Oracle Loader for Hadoop
      • Oracle Database Privileges for OLH Users
      • Provide Support for Offline Database Mode
      • Use Oracle Loader for Hadoop on a Secure Hadoop Cluster
    • Oracle Shell for Hadoop Loaders Setup
      • Install Oracle Shell for Hadoop Loaders on a Hadoop Node
      • Oracle Database Privileges for OHSH Users
      • Configure OHSH to Enable Job Monitoring
    • Oracle XQuery for Hadoop Setup
      • Software Requirements
      • Install Oracle XQuery for Hadoop
      • Troubleshoot the File Paths
      • Configure Oozie for the Oracle XQuery for Hadoop Action
    • Oracle R Advanced Analytics for Hadoop Setup
      • Install the Software on Hadoop
        • Software Requirements for a Third-Party Hadoop Cluster
        • Install Sqoop on a Third-Party Hadoop Cluster
        • Install Hive on a Third-Party Hadoop Cluster
        • Install R on a Hadoop Client
        • Install R on a Third-Party Hadoop Cluster
        • Install the ORCH Package on a Third-Party Hadoop Cluster
      • Install Additional R Packages
      • Provide Remote Client Access to R Users
        • Software Requirements for Remote Client Access
        • Configure the Server as a Hadoop Client
        • Install Sqoop on a Hadoop Client
        • Install R on a Hadoop Client
        • Install the ORCH Package on a Hadoop Client
        • Install the Oracle R Enterprise Client Packages (Optional)
    • Oracle Data Integrator
    • Oracle Datasource for Apache Hadoop Setup
      • Configure HiveServer2
• Part II Oracle Database Connectors
  • 2 Oracle Shell for Hadoop Loaders
    • What is Oracle Shell for Hadoop Loaders?
    • Configure Oracle Shell for Hadoop Loaders (OHSH)
    • Get Started with Oracle Shell for Hadoop Loaders
    • Load an Oracle Database Table
  • 3 Oracle SQL Connector for Hadoop Distributed File System
    • About Oracle SQL Connector for HDFS
    • Interfaces to Oracle SQL Connector for HDFS
    • Getting Started With Oracle SQL Connector for HDFS
    • Configure Your System for Oracle SQL Connector for HDFS
    • Use Oracle SQL Connector for HDFS with Oracle Big Data Appliance and Oracle Exadata
    • Use the ExternalTable Command-Line Tool
      • About ExternalTable
      • ExternalTable Command-Line Tool Syntax
    • Create External Tables
      • Create External Tables with the ExternalTable Tool
      • Create External Tables from Data Pump Format Files
        • Required Properties
        • Optional Properties
        • Defining Properties in XML Files for Data Pump Format Files
        • Example
      • Create External Tables from Hive Tables
        • Hive Table Requirements
        • Data Type Mappings
        • Required Properties
        • Optional Properties
        • Defining Properties in XML Files for Hive Tables
        • Example
        • Creating External Tables from Partitioned Hive Tables
          • Database Objects that Support Access to Partitioned Hive Tables
          • Querying the Metadata Table
          • Creating UNION ALL Views for Querying
          • Error Messages
          • Dropping Dangling Objects
      • Create External Tables from Delimited Text Files
        • Data Type Mappings
        • Required Properties
        • Optional Properties
        • Defining Properties in XML Files for Delimited Text Files
        • Example
      • Create External Tables in SQL
    • Update External Tables
      • ExternalTable Syntax for Publish
      • ExternalTable Example for Publish
    • Explore External Tables and Location Files
      • ExternalTable Syntax for Describe
      • ExternalTable Example for Describe
    • Drop Database Objects Created by Oracle SQL Connector for HDFS
      • ExternalTable Syntax for Drop
      • ExternalTable Example for Drop
    • More About External Tables Generated by the ExternalTable Tool
      • About Configurable Column Mappings
        • Default Column Mappings
        • All Column Overrides
        • One Column Overrides
        • Mapping Override Examples
      • What Are Location Files?
      • Enable Parallel Processing
        • Set Up Degree of Parallelism
      • Location File Management
      • Location File Names
    • Configure Oracle SQL Connector for HDFS
      • Create a Configuration File
      • Oracle SQL Connector for HDFS Configuration Property Reference
    • Performance Tips for Querying Data in HDFS
  • 4 Oracle Loader for Apache Hadoop
    • What Is Oracle Loader for Hadoop?
    • Interfaces to Oracle Loader For Hadoop
    • Use Oracle Loader for Hadoop With the Hadoop Command Line Utility
      • Interfaces to Oracle Loader for Hadoop
        • Online Database Mode
        • Offline Database Mode
      • Create the Target Table
        • Supported Data Types for Target Tables
        • Supported Partitioning Strategies for Target Tables
        • Compression
      • Create a Job Configuration File
      • Establish Secure Connections to Oracle Database Using SSL and Oracle Wallet
        • Use Oracle Wallets
        • Use JDBC SSL
          • Connect to Oracle Database Using JDBC SSL
        • Generate the Target Table Metadata for Offline Database Mode
      • About Input Formats
        • Delimited Text Input Format
          • About DelimitedTextInputFormat
          • Required Configuration Properties
          • Optional Configuration Properties
        • Complex Text Input Formats
          • About RegexInputFormat
          • Required Configuration Properties
          • Optional Configuration Properties
        • Hive Table Input Format
          • About HiveToAvroInputFormat
          • Required Configuration Properties
          • Optional Configuration Properties
        • Avro Input Format
          • Configuration Properties
        • Oracle NoSQL Database Input Format
          • About KVAvroInputFormat
          • Required Configuration Properties
        • Custom Input Formats
          • About Implementing a Custom Input Format
          • About Error Handling
          • Supporting Data Sampling
          • InputFormat Source Code Example
      • Mapping Input Fields to Target Table Columns
        • Automatic Mapping
        • Manual Mapping
        • Manual Mapping: Examples
      • About Output Formats
        • JDBC Output Format
          • About JDBCOutputFormat
          • Configuration Properties
        • Oracle OCI Direct Path Output Format
          • About OCIOutputFormat
          • Configuration Properties
        • Delimited Text Output Format
          • About DelimitedTextOutputFormat
          • Configuration Properties
        • Oracle Data Pump Output Format
          • About DataPumpOutputFormat
      • Run a Loader Job
        • Specify Hive Input Format JAR Files
        • Specify Oracle NoSQL Database Input Format JAR Files
        • Job Reporting
      • Handling Rejected Records
        • Log Rejected Records in Bad Files
        • Set a Job Reject Limit
      • Balancing Loads When Loading Data into Partitioned Tables
        • Use the Sampling Feature
        • Tuning Load Balancing
        • Tuning Sampling Behavior
        • When Does Oracle Loader for Hadoop Use the Sampler's Partitioning Scheme?
        • Resolve Memory Issues
        • What Happens When a Sampling Feature Property Has an Invalid Value?
      • Optimize Communications Between Oracle Engineered Systems
    • Oracle Loader for Hadoop Configuration Property Reference
• Part III Oracle XQuery for Apache Hadoop
  • 5 Using Oracle XQuery for Apache Hadoop
    • What Is Oracle XQuery for Hadoop?
    • Get Started With Oracle XQuery for Hadoop
      • Basic Steps
      • Example: Hello World!
    • About the Oracle XQuery for Hadoop Functions
      • About the Adapters
      • About Other Modules for Use With Oracle XQuery for Hadoop
    • Create an XQuery Transformation
      • XQuery Transformation Requirements
      • About XQuery Language Support
      • Accessing Data in the Hadoop Distributed Cache
      • Call Custom Java Functions from XQuery
      • Access User-Defined XQuery Library Modules and XML Schemas
      • XQuery Transformation Examples
    • Run Queries
      • Oracle XQuery for Hadoop Options
      • Generic Options
      • About Running Queries Locally
    • Run Queries from Apache Oozie
      • Use Oozie with Oracle XQuery for Hadoop Action
      • Supported XML Elements
      • Example: Hello World
    • Oracle XQuery for Hadoop Configuration Properties
  • 6 Oracle XQuery for Apache Hadoop Reference
    • Avro File Adapter
      • Built-in Functions for Reading Avro Files
        • avro:collection-avroxml
        • avro:get
      • Custom Functions for Reading Avro Container Files
      • Custom Functions for Writing Avro Files
      • Examples of Avro File Adapter Functions
      • About Converting Values Between Avro and XML
        • Reading Avro as XML
          • Reading Records
          • Reading Maps
          • Reading Arrays
          • Reading Unions
          • Reading Primitives
        • Writing XML as Avro
          • Writing Records
          • Writing Maps
          • Writing Arrays
          • Writing Unions
          • Writing Primitives
    • JSON File Adapter
      • Built-in Functions for Reading JSON
        • json:collection-jsonxml
        • json:parse-as-xml
        • json:get
      • Custom Functions for Reading JSON Files
      • Examples of JSON Functions
      • JSON File Adapter Configuration Properties
      • About Converting JSON Data Formats to XML
        • About Converting JSON Objects to XML
        • About Converting JSON Arrays to XML
        • About Converting Other JSON Types
    • Oracle Database Adapter
      • Custom Functions for Writing to Oracle Database
      • Examples of Oracle Database Adapter Functions
      • Oracle Loader for Hadoop Configuration Properties and Corresponding %oracle-property Annotations
    • Oracle NoSQL Database Adapter
      • Prerequisites for Using the Oracle NoSQL Database Adapter
      • Built-in Functions for Reading from and Writing to Oracle NoSQL Database
        • kv:collection-text
        • kv:collection-avroxml
        • kv:collection-xml
        • kv:collection-binxml
        • kv:collection-tika
        • kv:put-text
        • kv:put-xml
        • kv:put-binxml
        • kv:get-text
        • kv:get-avroxml
        • kv:get-xml
        • kv:get-binxml
        • kv:get-tika
        • kv:key-range
        • kv:key-range
      • Built-in Functions for Reading from and Writing to Oracle NoSQL Database using Table API
        • kv-table:collection-jsontext
        • kv-table:get-jsontext
        • kv-table:put-jsontext
      • Built-in Functions for Reading from and Writing to Oracle NoSQL Database using Large Object API
        • kv-lob:get-text
        • kv-lob:get-xml
        • kv-lob:get-binxml
        • kv-lob:get-tika
        • kv-lob:put-text
        • kv-lob:put-xml
        • kv-lob:put-binxml
      • Custom Functions for Reading Values from Oracle NoSQL Database
      • Custom Functions for Retrieving Single Values from Oracle NoSQL Database
      • Custom Functions for Reading Values from Oracle NoSQL Database using Table API
      • Custom Functions for Reading Single Row from Oracle NoSQL Database using Table API
      • Custom Functions for Retrieving Single Values from Oracle NoSQL Database using Large Object API
      • Custom Functions for Writing to Oracle NoSQL Database
      • Custom Functions for Writing Values to Oracle NoSQL Database using Table API
      • Custom Functions for Writing Values to Oracle NoSQL Database using Large Object API
      • Examples of Oracle NoSQL Database Adapter Functions
      • Oracle NoSQL Database Adapter Configuration Properties
    • Sequence File Adapter
      • Built-in Functions for Reading and Writing Sequence Files
        • seq:collection
        • seq:collection-xml
        • seq:collection-binxml
        • seq:collection-tika
        • seq:put
        • seq:put-xml
        • seq:put-binxml
      • Custom Functions for Reading Sequence Files
      • Custom Functions for Writing Sequence Files
      • Examples of Sequence File Adapter Functions
    • Solr Adapter
      • Prerequisites for Using the Solr Adapter
        • Configuration Settings
        • Example Query Using the Solr Adapter
      • Built-in Functions for Loading Data into Solr Servers
        • solr:put
      • Custom Functions for Loading Data into Solr Servers
      • Examples of Solr Adapter Functions
      • Solr Adapter Configuration Properties
    • Text File Adapter
      • Built-in Functions for Reading and Writing Text Files
        • text:collection
        • text:collection-xml
        • text:put
        • text:put-xml
        • text:trace
      • Custom Functions for Reading Text Files
      • Custom Functions for Writing Text Files
      • Examples of Text File Adapter Functions
    • Tika File Adapter
      • Built-in Library Functions for Parsing Files with Tika
        • tika:collection
        • tika:parse
      • Custom Functions for Parsing Files with Tika
      • Tika Parser Output Format
      • Tika Adapter Configuration Properties
      • Examples of Tika File Adapter Functions
    • XML File Adapter
      • Built-in Functions for Reading XML Files
        • xmlf:collection (Single Task)
        • xmlf:collection-multipart (Single Task)
        • xmlf:collection (Multiple Tasks)
      • Custom Functions for Reading XML Files
      • Examples of XML File Adapter Functions
    • Utility Module
      • Oracle XQuery Functions for Duration, Date, and Time
        • ora-fn:date-from-string-with-format
        • ora-fn:date-to-string-with-format
        • ora-fn:dateTime-from-string-with-format
        • ora-fn:dateTime-to-string-with-format
        • ora-fn:time-from-string-with-format
        • ora-fn:time-to-string-with-format
        • Format Argument
        • Locale Argument
      • Oracle XQuery Functions for Strings
        • ora-fn:pad-left
        • ora-fn:pad-right
        • ora-fn:trim
        • ora-fn:trim-left
        • ora-fn:trim-right
    • Hadoop Module
      • Built-in Functions for Using Hadoop
        • oxh:find
        • oxh:increment-counter
        • oxh:println
        • oxh:println-xml
        • oxh:property
    • Serialization Annotations
  • 7 Oracle XML Extensions for Hive
    • What are the XML Extensions for Hive?
    • Use the Hive Extensions From the Command Line
    • Use the Hive Extensions in HiveServer2
    • About the Hive Functions
    • Permanently Declaring the Hive Functions
    • Create XML Tables
      • Hive CREATE TABLE Syntax for XML Tables
      • CREATE TABLE Configuration Properties
      • CREATE TABLE Examples
        • Syntax Example
        • Simple Examples
        • OpenStreetMap Examples
    • Oracle XML Functions for Hive Reference
      • Data Type Conversions
      • Hive Access to External Files
    • Online Documentation of Functions
    • xml_exists
    • xml_query
    • xml_query_as_primitive
    • xml_table
• Part IV Oracle R Advanced Analytics for Apache Hadoop
  • 8 Oracle R Advanced Analytics for Apache Hadoop
    • About Oracle R Advanced Analytics for Hadoop
      • Oracle R Advanced Analytics for Hadoop Architecture
      • Oracle R Advanced Analytics for Hadoop packages and functions
      • Oracle R Advanced Analytics for Hadoop APIs
      • Inputs to Oracle R Advanced Analytics for Hadoop
    • Access to HDFS Files
    • Access to Apache Hive
      • ORCH Functions for Hive
      • ORE Functions for Hive
      • Generic R Functions Supported in Hive
      • Support for Hive Data Types
      • Usage Notes for Hive Access
      • Example: Loading Hive Tables into Oracle R Advanced Analytics for Hadoop
    • Access to Oracle Database
      • Usage Notes for Oracle Database Access
      • Scenario for Using Oracle R Advanced Analytics for Hadoop with Oracle R Enterprise
    • Oracle R Advanced Analytics for Hadoop Functions
      • Native Analytical Functions
      • Using the Hadoop Distributed File System (HDFS)
      • Using Apache Hive
      • Using Aggregate Functions in Hive
      • Making Database Connections
      • Copying Data and Working with HDFS Files
      • Converting to R Data Types
      • Using MapReduce
      • Debugging Scripts
    • Demos of Oracle R Advanced Analytics for Hadoop Functions
    • Security Notes for Oracle R Advanced Analytics for Hadoop
• Part V Oracle DataSource for Apache Hadoop
  • 9 Oracle DataSource for Apache Hadoop (OD4H)
    • Operational Data, Big Data and Requirements
    • Overview of Oracle DataSource for Apache Hadoop (OD4H)
      • Opportunity with Hadoop 2.x
      • Oracle Tables as Hadoop Data Source
      • External Tables
        • TBLPROPERTIES
        • SERDE PROPERTIES
      • List of jars in the OD4H package
    • How does OD4H work?
      • Create a new Oracle Database Table or Reuse an Existing Table
      • Hive DDL
      • Create External Tables in Hive
    • Features of OD4H
      • Performance And Scalability Features
        • Splitters
        • Choosing a Splitter
        • Predicate Pushdown
        • Projection Pushdown
        • Partition Pruning
      • Smart Connection Management
      • Security Features
        • Improved Authentication
    • Use HiveQL with OD4H
    • Use Spark SQL with OD4H
    • Writing Back to Oracle Database
• Appendices
  • A OraLoaderMetadata Utility
  • B Using Oracle's Hive Storage Handler for Kafka to Create a Hive External Table for Kafka Topics
  • C Apache License
    • Apache Licensed Code
  • D Oracle Big Data Connectors Accessibility Recommendations
    • Tips on Using Screen Readers and Braille Displays
    • Tips on Using Screen Magnifiers
  • E Additional Big Data Connector Resources
  • F Recent Change History
    • Changes in Oracle Big Data Connectors Release 4.12
• Index