oracle.spatial.hadoop.vector.mapred.job

Class Partitioning

java.lang.Object

org.apache.hadoop.conf.Configured

oracle.spatial.hadoop.vector.mapred.job.BaseJob<java.lang.Object,java.lang.Object>

oracle.spatial.hadoop.vector.mapred.job.MultipleInputsJob

oracle.spatial.hadoop.vector.mapred.job.Partitioning

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

Direct Known Subclasses:

Partitioning

public class Partitioning
extends MultipleInputsJob

This class can be used to run or configure a job for spatially partitioning one or more data sets.
Partitioning can be performed using only a fraction of the input data size by setting the setSamplingRatio(double) property (by default it is set to 0.1).
The job's output is a partitioning file, a GeoJSON file containing the generated partitions with information such as the partition geometry and number of elements in the partition.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	defaultNumReduceTasks
protected static java.lang.String	HV_RESULT_FOLDER
protected org.apache.hadoop.fs.Path	hvResultPath
static java.lang.String	PARTITION_RESULT_FILE
protected org.apache.hadoop.fs.Path	partitionsPath
protected static java.lang.String	SAMPLE_FILE_NAME
protected org.apache.hadoop.fs.Path	samplePath
protected double	samplingRatio

Fields inherited from class oracle.spatial.hadoop.vector.mapred.job.MultipleInputsJob

inputDataSets, miConf

Fields inherited from class oracle.spatial.hadoop.vector.mapred.job.BaseJob

argsp, inputDataSet, jarClass, jobRegEntryPath, proxyIDS

Constructor Summary

Constructors

Constructor and Description
Partitioning()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	clean(org.apache.hadoop.conf.Configuration conf)
void	configure(org.apache.hadoop.mapred.JobConf jobConf) Validates and adds the current parameters to the job configuration
protected AbstractInputDataSet	createSampleInputDataSet(org.apache.hadoop.conf.Configuration conf) Samples the input data and create a file containing the sampled geometries
protected void	defineGlobalBounds() Defines the dimension boundaries for the partitioning space based on the SpatialOperationConfig and the SpatialConfig defined for each input data set.
java.lang.String	getCmdOptions() Gets a description of the arguments expected from command line.
java.util.Map<java.lang.String,java.lang.Object>	getCurrentCmdArgs(org.apache.hadoop.conf.Configuration conf) Returns the current driver properties in a map where each key-value is a name and value of a command line argument.
protected void	getCurrentCmdArgsAsString(java.util.Map<java.lang.String,java.lang.Object> cmdArgs, java.lang.StringBuilder buff)
protected org.apache.hadoop.fs.Path	getHVResultPartPath(org.apache.hadoop.fs.Path sampleDir, org.apache.hadoop.conf.Configuration conf)
protected InputDataSetCmdArgsParserHandler	getInputDataSetCmdParserHandler(org.apache.hadoop.conf.Configuration conf) Gets the current instance of InputDataSetCmdArgsParserHandler used to parse command line parameters for the input data set
protected InputDataSetConfiguratorHandler	getInputDataSetConfiguratorHandler(org.apache.hadoop.conf.Configuration conf) Returns the current instance of InputDataSetConfiguratorHandler used to configure the input data set
org.apache.hadoop.fs.Path	getPartitionsPath()
protected long	getPathsLength(org.apache.hadoop.fs.Path[] paths, org.apache.hadoop.conf.Configuration conf) Deprecated.
double	getSamplingRatio() Gets the ratio of the sample size to the input data size
static void	main(java.lang.String[] args)
void	processArgs(java.lang.String[] args, org.apache.hadoop.conf.Configuration conf) Extracts and validates arguments from the command line
int	run(java.lang.String[] args)
boolean	runFullPartitioningProcess(org.apache.hadoop.mapred.JobConf jobConf) Runs the full partitioning process.
void	setSamplingRatio(double samplingRatio) Sets the ratio of the sample size to the input data size so only a fraction of the whole input data is used for partitioning.

Methods inherited from class oracle.spatial.hadoop.vector.mapred.job.MultipleInputsJob

addInputDataSet, asMultiInputDataSet, configureInputs, configureInputs, getInputListCmdOptions, getInputs, getMultipleInputDataSetsParams, removeInputDataSet, setInputDataSets, updateInputDataSet

Methods inherited from class oracle.spatial.hadoop.vector.mapred.job.BaseJob

addJobRegistryEntry, addJobRegistryEntry, addJobRegistryEntry, configure, createJob, createJob, createJob, createJob, createJobConf, createJobConf, createJobConf, getCmdOptionsWithInputDataSets, getCmdOptionsWithInputDataSets, getCurrentCmdArgsAsString, getInput, getInputDataSet, getInputFormatClass, getJarClass, getOutput, getRecordInfoProviderClass, getSpatialConfig, runJob, runJob, setInput, setInputDataSet, setInputFormatClass, setJarClass, setOutput, setRecordInfoProviderClass, setSpatialConfig

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

PARTITION_RESULT_FILE

public static final java.lang.String PARTITION_RESULT_FILE

See Also:

Constant Field Values

SAMPLE_FILE_NAME

protected static final java.lang.String SAMPLE_FILE_NAME

See Also:

Constant Field Values

HV_RESULT_FOLDER

protected static final java.lang.String HV_RESULT_FOLDER

See Also:

Constant Field Values

samplingRatio

protected double samplingRatio

defaultNumReduceTasks

protected int defaultNumReduceTasks

samplePath

protected org.apache.hadoop.fs.Path samplePath

hvResultPath

protected org.apache.hadoop.fs.Path hvResultPath

partitionsPath

protected org.apache.hadoop.fs.Path partitionsPath

Constructor Detail

Partitioning

public Partitioning()

Method Detail

getInputDataSetConfiguratorHandler

protected InputDataSetConfiguratorHandler getInputDataSetConfiguratorHandler(org.apache.hadoop.conf.Configuration conf)

Description copied from class: BaseJob

Returns the current instance of InputDataSetConfiguratorHandler used to configure the input data set

Specified by:

getInputDataSetConfiguratorHandler in class BaseJob<java.lang.Object,java.lang.Object>

Parameters:

conf - a job configuration

Returns:

an instance of InputDataSetConfiguratorHandler

getInputDataSetCmdParserHandler

protected InputDataSetCmdArgsParserHandler getInputDataSetCmdParserHandler(org.apache.hadoop.conf.Configuration conf)

Description copied from class: BaseJob

Gets the current instance of InputDataSetCmdArgsParserHandler used to parse command line parameters for the input data set

Specified by:

getInputDataSetCmdParserHandler in class BaseJob<java.lang.Object,java.lang.Object>

Parameters:

conf - a job configuration

Returns:

an instance of InputDataSetCmdArgsParserHandler

getPartitionsPath

public org.apache.hadoop.fs.Path getPartitionsPath()

setSamplingRatio

public void setSamplingRatio(double samplingRatio)

Sets the ratio of the sample size to the input data size so only a fraction of the whole input data is used for partitioning.

Parameters:

samplingRatio -

getSamplingRatio

public double getSamplingRatio()

Gets the ratio of the sample size to the input data size

Returns:

processArgs

public void processArgs(java.lang.String[] args,
               org.apache.hadoop.conf.Configuration conf)
                 throws java.lang.Exception

Description copied from class: BaseJob

Extracts and validates arguments from the command line

Overrides:

processArgs in class MultipleInputsJob

Parameters:

args - arguments from the command line

conf - the job configuration

Throws:

java.lang.Exception

getCurrentCmdArgsAsString

protected void getCurrentCmdArgsAsString(java.util.Map<java.lang.String,java.lang.Object> cmdArgs,
                             java.lang.StringBuilder buff)

Overrides:

getCurrentCmdArgsAsString in class BaseJob<java.lang.Object,java.lang.Object>

getCurrentCmdArgs

public java.util.Map<java.lang.String,java.lang.Object> getCurrentCmdArgs(org.apache.hadoop.conf.Configuration conf)

Description copied from class: BaseJob

Returns the current driver properties in a map where each key-value is a name and value of a command line argument. By printing this information it is possible to know how to execute a similar job from command line

Overrides:

getCurrentCmdArgs in class BaseJob<java.lang.Object,java.lang.Object>

Parameters:

conf - a job configuration

Returns:

a map which entries are of type <String,String> or <String, Map<String, Object>> for multiple input data sets. For nested maps, the entry types are the same than the enclosing map

getCmdOptions

public java.lang.String getCmdOptions()

Description copied from class: BaseJob

Gets a description of the arguments expected from command line.

Overrides:

getCmdOptions in class BaseJob<java.lang.Object,java.lang.Object>

Returns:

a text description

configure

public void configure(org.apache.hadoop.mapred.JobConf jobConf)
               throws java.lang.Exception

Description copied from class: BaseJob

Validates and adds the current parameters to the job configuration

Overrides:

configure in class BaseJob<java.lang.Object,java.lang.Object>

Parameters:

jobConf - the job configuration

Throws:

java.lang.Exception

defineGlobalBounds

protected void defineGlobalBounds()

Defines the dimension boundaries for the partitioning space based on the SpatialOperationConfig and the SpatialConfig defined for each input data set.

createSampleInputDataSet

protected AbstractInputDataSet createSampleInputDataSet(org.apache.hadoop.conf.Configuration conf)
                                                 throws java.io.IOException

Samples the input data and create a file containing the sampled geometries

Parameters:

conf - the job configuration

Returns:

an AbstractInputDataSet pointing to the sampled data

Throws:

java.io.IOException

runFullPartitioningProcess

public boolean runFullPartitioningProcess(org.apache.hadoop.mapred.JobConf jobConf)
                                   throws java.lang.Exception

Runs the full partitioning process. It is not needed to call configure() when calling this method

Parameters:

jobConf -

Returns:

true if the job finished successfully

Throws:

java.lang.Exception

run

public int run(java.lang.String[] args)
        throws java.lang.Exception

Throws:

java.lang.Exception

getPathsLength

@Deprecated
protected long getPathsLength(org.apache.hadoop.fs.Path[] paths,
                             org.apache.hadoop.conf.Configuration conf)
                       throws java.io.IOException

Deprecated.

Throws:

java.io.IOException

getHVResultPartPath

protected org.apache.hadoop.fs.Path getHVResultPartPath(org.apache.hadoop.fs.Path sampleDir,
                                            org.apache.hadoop.conf.Configuration conf)
                                                 throws java.io.IOException

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Throws:

java.lang.Exception

clean

protected void clean(org.apache.hadoop.conf.Configuration conf)
              throws java.io.IOException

Throws:

java.io.IOException