Go to main content
1/15
Contents
List of Figures
List of Tables
Title and Copyright Information
Preface
Audience
Documentation Accessibility
Related Documents
Conventions
Changes in This Release for Oracle Big Data Spatial and Graph
In-Memory Analyst (PGX) - Related Changes
New PGX Built-in Algorithms for Cycle Detection
Temporal Data Types Support in PGX
PGX Java API Improvements
New Features in PGQL
PGX Loader Improvements
PGX Distributed Engine Improvements
PGX Deprecations
Spark Vector API Changes
Vector REST API Additions
SpatialViewer Changes
1
Big Data Spatial and Graph Overview
1.1
About Big Data Spatial and Graph
1.2
Spatial Features
1.3
Property Graph Features
1.3.1
Property Graph Sizing Recommendations
1.4
Multimedia Analytics Features
1.5
Installing Oracle Big Data Spatial and Graph on an Oracle Big Data Appliance
1.6
Installing and Configuring the Big Data Spatial Image Processing Framework
1.6.1
Getting and Compiling the Cartographic Projections Library
1.6.2
Installing the Image Processing Framework for Oracle Big Data Appliance Distribution
1.6.3
Installing the Image Processing Framework for Other Distributions (Not Oracle Big Data Appliance)
1.6.3.1
Prerequisites for Installing the Image Processing Framework for Other Distributions
1.6.3.2
Installing the Image Processing Framework for Other Distributions
1.6.4
Post-installation Verification of the Image Processing Framework
1.6.4.1
Image Loading Test Script
1.6.4.2
Image Processor Test Script (Mosaicking)
1.6.4.3
Single-Image Processor Test Script
1.6.4.4
Image Processor DEM Test Script
1.6.4.5
Multiple Raster Operation Test Script
1.7
Installing and Configuring the Big Data Spatial Image Server
1.7.1
Installing and Configuring the Image Server for Oracle Big Data Appliance
1.7.1.1
Prerequisites for Performing a Manual Installation
1.7.1.2
Installing Dependencies on the Image Server Web on an Oracle Big Data Appliance
1.7.1.3
Configuring the Environment for Big Data Appliance
1.7.2
Installing and Configuring the Image Server Web for Other Systems (Not Big Data Appliance)
1.7.2.1
Prerequisites for Installing the Image Server on Other Systems
1.7.2.2
Installing the Image Server Web on Other Systems
1.7.2.3
Configuring the Environment for Other Systems
1.7.3
Post-Installation Verification Example for the Image Server Console
1.7.3.1
Loading Images from the Local Server to the HDFS Hadoop Cluster
1.7.3.2
Creating a Mosaic Image and Catalog
1.7.3.3
Creating a Mosaic Directly from the Globe
1.7.3.4
Creating a Slope Image from the Globe
1.7.3.5
Creating an Image Using Multiple-Raster Algebra Operations
1.7.3.6
Removing Identical Rasters
1.7.4
Using the Provided Image Server Web Services
1.8
Installing the Oracle Big Data SpatialViewer Web Application
1.8.1
Assumptions for SpatialViewer
1.8.2
Installing SpatialViewer on Oracle Big Data Appliance
1.8.3
Installing SpatialViewer for Other Systems (Not Big Data Appliance)
1.8.4
Configuring SpatialViewer on Oracle Big Data Appliance
1.8.5
Configuring SpatialViewer for Other Systems (Not Big Data Appliance)
1.9
Installing Property Graph Support on a CDH Cluster or Other Hardware
1.9.1
Apache HBase Prerequisites
1.9.2
Property Graph Installation Steps
1.9.3
About the Property Graph Installation Directory
1.9.4
Optional Installation Task for In-Memory Analyst Use
1.9.4.1
Installing and Configuring Hadoop
1.9.4.2
Running the In-Memory Analyst on Hadoop
1.10
Installing and Configuring Multimedia Analytics Support
1.10.1
Assumptions and Libraries for Multimedia Analytics
1.10.2
Transcoding Software (Options)
2
Using Big Data Spatial and Graph with Spatial Data
2.1
About Big Data Spatial and Graph Support for Spatial Data
2.1.1
What is Big Data Spatial and Graph on Apache Hadoop?
2.1.2
Advantages of Oracle Big Data Spatial and Graph
2.1.3
Oracle Big Data Spatial Features and Functions
2.1.4
Oracle Big Data Spatial Files, Formats, and Software Requirements
2.2
Oracle Big Data Vector and Raster Data Processing
2.2.1
Oracle Big Data Spatial Raster Data Processing
2.2.2
Oracle Big Data Spatial Vector Data Processing
2.3
Oracle Big Data Spatial Hadoop Image Processing Framework for Raster Data Processing
2.3.1
Image Loader
2.3.2
Image Processor
2.3.3
Image Server
2.4
Loading an Image to Hadoop Using the Image Loader
2.4.1
Image Loading Job
2.4.2
Input Parameters
2.4.3
Output Parameters
2.5
Processing an Image Using the Oracle Spatial Hadoop Image Processor
2.5.1
Image Processing Job
2.5.1.1
Default Image Processing Job Flow
2.5.1.2
Multiple Raster Image Processing Job Flow
2.5.2
Input Parameters
2.5.2.1
Catalog XML Structure
2.5.2.2
Mosaic Definition XML Structure
2.5.3
Job Execution
2.5.4
Processing Classes and ImageBandWritable
2.5.4.1
Location of the Classes and Jar Files
2.5.5
Map Algebra Operations
2.5.6
Multiple Raster Algebra Operations
2.5.6.1
Basic Multiple Raster Algebra Operations
2.5.6.2
Complex Multiple Raster Algebra Operations
2.5.7
Output
2.6
Loading and Processing an Image Using the Oracle Spatial Hadoop Raster Processing API
2.7
Using the Oracle Spatial Hadoop Raster Simulator Framework to Test Raster Processing
2.8
Oracle Big Data Spatial Raster Processing for Spark
2.8.1
Spark Raster Loader
2.8.1.1
Input Parameters to the Spark Raster Loader
2.8.1.2
Expected Output of the Spark Raster Loader
2.8.2
Spark SQL Raster Processor
2.8.2.1
Input Parameters to the Spark SQL Raster Processor
2.8.2.2
Expected Output of the Spark SQL Raster Processor
2.8.3
Using the Spark Raster Processing API
2.8.3.1
Using the Spark Raster Loader API
2.8.3.2
Configuring for Using the Spark SQL Processor API
2.8.3.3
Creating the DataFrame
2.8.3.4
Using the Spark SQL UDF for Raster Algebra Operations
2.9
Oracle Big Data Spatial Vector Analysis
2.9.1
Multiple Hadoop API Support
2.9.2
Spatial Indexing
2.9.2.1
Spatial Indexing Class Structure
2.9.2.2
Configuration for Creating a Spatial Index
2.9.2.3
Spatial Index Metadata
2.9.2.4
Input Formats for a Spatial Index
2.9.2.5
Support for GeoJSON and Shapefile Formats
2.9.2.6
Removing a Spatial Index
2.9.3
Using MVSuggest
2.9.4
Spatial Filtering
2.9.4.1
Filtering Records
2.9.4.2
Filtering Using the Input Format
2.9.5
Classifying Data Hierarchically
2.9.5.1
Changing the Hierarchy Level Range
2.9.5.2
Controlling the Search Hierarchy
2.9.5.3
Using MVSuggest to Classify the Data
2.9.6
Generating Buffers
2.9.7
Spatial Binning
2.9.8
Spatial Clustering
2.9.9
Spatial Join
2.9.10
Spatial Partitioning
2.9.11
RecordInfoProvider
2.9.11.1
Sample RecordInfoProvider Implementation
2.9.11.2
LocalizableRecordInfoProvider
2.9.12
HierarchyInfo
2.9.12.1
Sample HierarchyInfo Implementation
2.9.13
Using JGeometry in MapReduce Jobs
2.9.14
Support for Different Data Sources
2.9.15
Job Registry
2.9.16
Tuning Performance Data of Job Running Times Using the Vector Analysis API
2.10
Oracle Big Data Spatial Vector Analysis for Spark
2.10.1
Spatial RDD (Resilient Distributed Dataset)
2.10.2
Spatial Transformations
2.10.2.1
Filter Transformation
2.10.2.2
FlatMap Transformation
2.10.2.3
Join Transformation
2.10.2.4
Controlling Spatial Evaluation
2.10.2.5
Spatially Enabled Transformations
2.10.3
Spatial Actions (MBR and NearestNeighbors)
2.10.4
Spatially Indexing a Spatial RDD
2.10.4.1
Spatial Partitioning of a Spatial RDD
2.10.4.2
Local Spatial Indexing of a Spatial RDD
2.10.5
Input Format Support
2.10.6
Spatial Spark SQL API
2.10.7
JDBC Data Sources for Spatial RDDs
2.11
Oracle Big Data Spatial Vector Hive Analysis
2.11.1
HiveRecordInfoProvider
2.11.2
Using the Hive Spatial API
2.11.3
Using Spatial Indexes in Hive
2.12
Using the Oracle Big Data SpatialViewer Web Application
2.12.1
Creating a Hadoop Spatial Index Using SpatialViewer
2.12.2
Exploring the Hadoop Indexed Spatial Data
2.12.3
Creating a Spark Spatial Index Using SpatialViewer
2.12.4
Exploring the Spark Indexed Spatial Data
2.12.5
Running a Categorization Job Using SpatialViewer
2.12.6
Viewing the Categorization Results
2.12.7
Saving Categorization Results to a File
2.12.8
Creating and Deleting Templates
2.12.9
Configuring Templates
2.12.10
Running a Clustering Job Using SpatialViewer
2.12.11
Viewing the Clustering Results
2.12.12
Saving Clustering Results to a File
2.12.13
Running a Binning Job Using SpatialViewer
2.12.14
Viewing the Binning Results
2.12.15
Saving Binning Results to a File
2.12.16
Running a Job to Create an Index Using the Command Line
2.12.17
Running a Job to Create a Categorization Result
2.12.18
Running a Job to Create a Clustering Result
2.12.19
Running a Job to Create a Binning Result
2.12.20
Running a Job to Perform Spatial Filtering
2.12.21
Running a Job to Get Location Suggestions
2.12.22
Running a Job to Perform a Spatial Join
2.12.23
Running a Job to Perform Partitioning
2.12.24
Using Multiple Inputs
2.12.25
Loading Images from the Local Server to the HDFS Hadoop Cluster
2.12.26
Visualizing Rasters in the Globe
2.12.27
Processing a Raster or Multiple Rasters with the Same MBR
2.12.28
Creating a Mosaic Directly from the Globe
2.12.29
Adding Operations for Raster Processing
2.12.30
Creating a Slope Image from the Globe
2.12.31
Changing the Image File Format from the Globe
2.13
Using Oracle Big Data Spatial and Graph Image Server Console
2.13.1
Loading Images to an HDFS Hadoop Cluster to Create a Mosaic
3
Integrating Big Data Spatial and Graph with Oracle Database
3.1
Using Oracle SQL Connector for HDFS with Delimited Text Files
3.2
Using Oracle SQL Connector for HDFS with Hive Tables
3.3
Using Oracle SQL Connector for HDFS with Files Generated by Oracle Loader for Hadoop
3.3.1
Creating HDFS Data Pump Files or Delimited Text Files
3.3.2
Creating the SQL Connector for HDFS
3.4
Integrating HDFS Spatial Data with Oracle Database Using Oracle Big Data SQL
3.4.1
Creating Oracle External Tables for HDFS Files with Big Data SQL
3.4.2
Creating Oracle External Tables Using Hive Tables with Big Data SQL
4
Configuring Property Graph Support
4.1
Tuning Apache HBase for Use with Property Graphs
4.1.1
Modifying the Apache HBase Configuration
4.1.2
Modifying the Java Memory Settings
4.2
Tuning Oracle NoSQL Database for Use with Property Graphs
5
Using Property Graphs in a Big Data Environment
5.1
About Property Graphs
5.1.1
What Are Property Graphs?
5.1.2
What Is Big Data Support for Property Graphs?
5.1.2.1
In-Memory Analyst
5.1.2.2
Data Access Layer
5.1.2.3
Storage Management
5.1.2.4
RESTful Web Services
5.2
About Property Graph Data Formats
5.2.1
GraphML Data Format
5.2.2
GraphSON Data Format
5.2.3
GML Data Format
5.2.4
Oracle Flat File Format
5.3
Getting Started with Property Graphs
5.4
Using Java APIs for Property Graph Data
5.4.1
Overview of the Java APIs
5.4.1.1
Oracle Big Data Spatial and Graph Java APIs
5.4.1.2
TinkerPop Blueprints Java APIs
5.4.1.3
Apache Hadoop Java APIs
5.4.1.4
Oracle NoSQL Database Java APIs
5.4.1.5
Apache HBase Java APIs
5.4.2
Parallel Loading of Graph Data
5.4.2.1
Parallel Data Loading Using Partitions
5.4.2.2
Parallel Data Loading Using Fine-Tuning
5.4.2.3
Parallel Data Loading Using Multiple Files
5.4.2.4
Parallel Retrieval of Graph Data
5.4.2.5
Using an Element Filter Callback for Subgraph Extraction
5.4.2.6
Using Optimization Flags on Reads over Property Graph Data
5.4.2.7
Adding and Removing Attributes of a Property Graph Subgraph
5.4.2.8
Getting Property Graph Metadata
5.4.3
Opening and Closing a Property Graph Instance
5.4.3.1
Using Oracle NoSQL Database
5.4.3.2
Using Apache HBase
5.4.4
Creating Vertices
5.4.5
Creating Edges
5.4.6
Deleting Vertices and Edges
5.4.7
Reading a Graph from a Database into an Embedded In-Memory Analyst
5.4.8
Specifying Labels for Vertices
5.4.9
Building an In-Memory Graph
5.4.10
Dropping a Property Graph
5.4.10.1
Using Oracle NoSQL Database
5.4.10.2
Using Apache HBase
5.5
Managing Text Indexing for Property Graph Data
5.5.1
Configuring a Text Index for Property Graph Data
5.5.2
Using Automatic Indexes for Property Graph Data
5.5.3
Using Manual Indexes for Property Graph Data
5.5.4
Executing Search Queries Over Property Graph Text Indexes
5.5.5
Handling Data Types
5.5.5.1
Appending Data Type Identifiers on Apache Lucene
5.5.5.2
Appending Data Type Identifiers on SolrCloud
5.5.6
Uploading a Collection's SolrCloud Configuration to Zookeeper
5.5.7
Updating Configuration Settings on Text Indexes for Property Graph Data
5.5.8
Using Parallel Query on Text Indexes for Property Graph Data
5.5.9
Using Native Query Objects on Text Indexes for Property Graph Data
5.5.10
Using Native Query Results on Text Indexes for Property Graph Data
5.6
Querying Property Graph Data Using PGQL
5.7
Using Apache Spark with Property Graph Data
5.7.1
Using Apache Spark with Property Graph Data in Apache HBase
5.7.2
Integrating Apache Spark with Property Graph Data Stored in Oracle NoSQL Database
5.8
Support for Secure Oracle NoSQL Database
5.9
Implementing Security on Graphs Stored in Apache HBase
5.10
Using the Groovy Shell with Property Graph Data
5.11
REST Support for Property Graph Data
5.11.1
Building the REST Web Application Archive (WAR) File
5.11.2
Deploying the RESTful Property Graph Web Service
5.11.2.1
RESTful Property Graph Service Configuration File (rexster.xml)
5.11.3
Property Graph REST API Operations Information
5.11.3.1
GET Operations (Property Graphs)
5.11.3.2
POST Operations (Property Graphs)
5.11.3.3
PUT Operations (Property Graphs)
5.11.3.4
DELETE Operations (Property Graphs)
5.12
Exploring the Sample Programs
5.12.1
About the Sample Programs
5.12.2
Compiling and Running the Sample Programs
5.12.3
About the Example Output
5.12.4
Example: Creating a Property Graph
5.12.5
Example: Dropping a Property Graph
5.12.6
Examples: Adding and Dropping Vertices and Edges
5.13
Oracle Flat File Format Definition
5.13.1
About the Property Graph Description Files
5.13.2
Vertex File
5.13.3
Edge File
5.13.4
Encoding Special Characters
5.13.5
Example Property Graph in Oracle Flat File Format
5.13.6
Converting an Oracle Database Table to an Oracle-Defined Property Graph Flat File
5.13.7
Converting CSV Files for Vertices and Edges to Oracle-Defined Property Graph Flat Files
5.13.7.1
Vertices: Converting a CSV File to Oracle-Defined Flat File Format (.opv)
5.13.7.2
Edges: Converting a CSV File to Oracle-Defined Flat File Format (.ope)
5.13.7.3
Vertices and Edges: Converting a Single CSV File Containing Both Vertices and Edges Data into a Pair of Graph Flat Files
5.14
Example Python User Interface
5.15
Example iPython Notebooks User Interface
6
Using the In-Memory Analyst (PGX)
6.1
Reading a Graph into Memory
6.1.1
Connecting to an In-Memory Analyst Server Instance
6.1.2
Using the Shell Help
6.1.3
Providing Graph Metadata in a Configuration File
6.1.4
Reading Graph Data into Memory
6.1.4.1
Read a Graph Stored in Apache HBase into Memory
6.1.4.2
Read a Graph Stored in Oracle NoSQL Database into Memory
6.1.4.3
Read a Graph Stored in the Local File System into Memory
6.2
Configuring the In-Memory Analyst
6.2.1
Specifying the Configuration File to the In-Memory Analyst
6.3
Reading Custom Graph Data
6.3.1
Creating a Simple Graph File
6.3.2
Adding a Vertex Property
6.3.3
Using Strings as Vertex Identifiers
6.3.4
Adding an Edge Property
6.4
Storing Graph Data on Disk
6.4.1
Storing the Results of Analysis in a Vertex Property
6.4.2
Storing a Graph in Edge-List Format on Disk
6.5
Executing Built-in Algorithms
6.5.1
About the In-Memory Analyst
6.5.2
Running the Triangle Counting Algorithm
6.5.3
Running the Pagerank Algorithm
6.6
Creating Subgraphs
6.6.1
About Filter Expressions
6.6.2
Using a Simple Edge Filter to Create a Subgraph
6.6.3
Using a Simple Vertex Filter to Create a Subgraph
6.6.4
Using a Complex Filter to Create a Subgraph
6.6.5
Combining Expression Filters
6.6.6
Using an Expression Filter to Create a Set of Vertices or Edges
6.6.7
Using a Vertex Set to Create a Bipartite Subgraph
6.7
Using Pattern-Matching Queries with Graphs
6.7.1
Example: The Enemy of My Enemy is My Friend
6.7.2
Example: Top 10 Most Collaborative People
6.7.3
Example: Transitive Connectivity Between Electrical Devices
6.8
Starting the In-Memory Analyst Server
6.8.1
Configuring the In-Memory Analyst Server
6.9
Deploying to Jetty
6.10
Deploying to Apache Tomcat
6.11
Deploying to Oracle WebLogic Server
6.11.1
Installing Oracle WebLogic Server
6.11.2
Deploying the In-Memory Analyst
6.11.3
Verifying That the Server Works
6.12
Connecting to the In-Memory Analyst Server
6.12.1
Connecting with the In-Memory Analyst Shell
6.12.1.1
About Logging HTTP Requests
6.12.2
Connecting with Java
6.12.3
Connecting with JavaScript
6.13
Using the In-Memory Analyst in Distributed Mode
6.14
Reading and Storing Data in HDFS
6.14.1
Reading Data from HDFS
6.14.2
Storing Graph Snapshots in HDFS
6.14.3
Compiling and Running a Java Application in Hadoop
6.15
Running the In-Memory Analyst as a YARN Application
6.15.1
Starting and Stopping In-Memory Analyst Services
6.15.1.1
Configuring the In-Memory Analyst YARN Client
6.15.1.2
Starting a New In-Memory Analyst Service
6.15.1.3
About Long-Running In-Memory Analyst Services
6.15.1.4
Stopping In-Memory Analyst Services
6.15.2
Connecting to In-Memory Analyst Services
6.15.3
Monitoring In-Memory Analyst Services
6.16
Using Oracle Two-Tables Relational Format
6.17
Using the In-Memory Analyst to Analyze Graph Data in Apache Spark
6.17.1
Controlling the Degree of Parallelism in Apache Spark
6.18
Using the In-Memory Analyst Zeppelin Interpreter
6.19
Using the In-Memory Analyst Enterprise Scheduler
6.19.1
Using Lambda Syntax with Execution Environments
7
Using Multimedia Analytics
7.1
About Multimedia Analytics
7.2
Processing Video and Image Data Stored in HDFS Using the Multimedia Analytics Framework
7.3
Processing Streaming Video Using the Multimedia Analytics Framework
7.4
Face Recognition Using the Multimedia Analytics Framework
7.4.1
Training to Detect Faces
7.4.2
Selecting Faces to be Used for Training
7.4.3
Detecting Faces in Videos
7.4.4
Detecting Faces in Images
7.4.5
Working with Oracle NoSQL Database
7.4.6
Working with Apache HBase
7.4.7
Examples and Training Materials for Detecting Faces
7.5
Configuration Properties for Multimedia Analytics
7.5.1
Configuration Properties for Processing Stored Videos and Images
7.5.2
Configuration Properties for Processing Streaming Video
7.5.3
Configuration Properties for Training Images for Face Recognition
7.6
Using the Multimedia Analytics Framework with Third-Party Software
7.7
Displaying Images in Output
A
Third-Party Licenses for Bundled Software
A.1
Apache Licensed Code
A.2
ANTLR 3
A.3
AOP Alliance
A.4
Apache Commons CLI
A.5
Apache Commons Codec
A.6
Apache Commons Collections
A.7
Apache Commons Configuration
A.8
Apache Commons IO
A.9
Apache Commons Lang
A.10
Apache Commons Logging
A.11
Apache Commons VFS
A.12
Apache fluent
A.13
Apache Groovy
A.14
Apache htrace
A.15
Apache HTTP Client
A.16
Apache HTTPComponents Core
A.17
Apache Jena
A.18
Apache Log4j
A.19
Apache Lucene
A.20
Apache Tomcat
A.21
Apache Xerces2
A.22
Apache xml-commons
A.23
Argparse4j
A.24
check-types
A.25
Cloudera CDH
A.26
cookie
A.27
Fastutil
A.28
functionaljava
A.29
GeoNames Data
A.30
Geospatial Data Abstraction Library (GDAL)
A.31
Google Guava
A.32
Google Guice
A.33
Google protobuf
A.34
int64-native
A.35
Jackson
A.36
Jansi
A.37
JCodec
A.38
Jettison
A.39
JLine
A.40
Javassist
A.41
json-bignum
A.42
Jung
A.43
Log4js
A.44
MessagePack
A.45
Netty
A.46
Node.js
A.47
node-zookeeper-client
A.48
OpenCV
A.49
rxjava-core
A.50
Slf4j
A.51
Spoofax
A.52
Tinkerpop Blueprints
A.53
Tinkerpop Gremlin
A.54
Tinkerpop Pipes
B
Hive Spatial Functions
B.1
ST_AnyInteract
B.2
ST_Area
B.3
ST_AsWKB
B.4
ST_AsWKT
B.5
ST_Buffer
B.6
ST_Contains
B.7
ST_ConvexHull
B.8
ST_Distance
B.9
ST_Envelope
B.10
ST_Geometry
B.11
ST_Inside
B.12
ST_Length
B.13
ST_LineString
B.14
ST_MultiLineString
B.15
ST_MultiPoint
B.16
ST_MultiPolygon
B.17
ST_Point
B.18
ST_Polygon
B.19
ST_Simplify
B.20
ST_SimplifyVW
B.21
ST_Volume
Scripting on this page enhances content navigation, but does not change the content in any way.