oracle.spatial.hadoop.vector.util

Class JobUtils

java.lang.Object

oracle.spatial.hadoop.vector.util.JobUtils

public class JobUtils
extends java.lang.Object

Contains convenience methods used among demo jobs

Constructor Summary

Constructors

Constructor and Description
JobUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static <K,V> java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>>	getMapredInputFormat(org.apache.hadoop.mapred.JobConf conf, java.lang.Class<?> rawClass) Checks if the given class is an org.apache.hadoop.mapred.InputFormat or an org.apache.hadoop.mapreduce.lib.input.FileInputFormat and returns the appropriate adapter if needed.
static org.apache.hadoop.fs.Path[]	getPaths(java.lang.String inStr, org.apache.hadoop.conf.Configuration conf)
static void	jobCleanup(org.apache.hadoop.conf.Configuration conf) Performs cleaning tasks once a job has finished.
static org.apache.hadoop.fs.Path	setupInputPath(java.lang.String inStr, org.apache.hadoop.mapreduce.Job job)
static org.apache.hadoop.fs.Path	setupInputPath(java.lang.String inStr, org.apache.hadoop.mapred.JobConf conf) A convenience method to infer a job's input provided by the user.
static org.apache.hadoop.fs.Path	setupOutputPath(java.lang.String outStr, org.apache.hadoop.fs.Path basePath, org.apache.hadoop.mapreduce.Job job)
static org.apache.hadoop.fs.Path	setupOutputPath(java.lang.String outStr, org.apache.hadoop.fs.Path basePath, org.apache.hadoop.mapred.JobConf conf) Convenience method for setting a job's output path.
static void	validateInputDataSet(InputDataSet inputDs, org.apache.hadoop.conf.Configuration conf)
static void	validateMapRedInputDataSet(InputDataSet inputDs, org.apache.hadoop.mapred.JobConf jobConf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JobUtils

public JobUtils()

Method Detail

setupInputPath

public static org.apache.hadoop.fs.Path setupInputPath(java.lang.String inStr,
                                                       org.apache.hadoop.mapred.JobConf conf)
                                                throws java.io.IOException

A convenience method to infer a job's input provided by the user. The input path is determined using the following criteria:

1. If the input is a comma-separated paths, it is set as is and the method returns null
2. If the input exists, it is set as is and the method returns the given input as a path
3. The input is considered a glob. If files were found the method returns the input glob as a path
4. The input's file name is considered a regular expression. A file pattern filter is set to use the regular expression and the method returns the input's parent folder as a path.

Parameters:

inStr - the input data location. It may be a file's path, a comma-separated paths, a glob expression or an expression of the form path/<filename as regex>, that is, a path where the file name is given by a regular expression.

conf - the job configuration

Returns:

a path in case of using a single path or base path or null in case of the input being a comma-separated list of paths

Throws:

java.io.IOException

setupInputPath

public static org.apache.hadoop.fs.Path setupInputPath(java.lang.String inStr,
                                                       org.apache.hadoop.mapreduce.Job job)
                                                throws java.io.IOException

Throws:

java.io.IOException

getPaths

public static org.apache.hadoop.fs.Path[] getPaths(java.lang.String inStr,
                                                   org.apache.hadoop.conf.Configuration conf)
                                            throws java.io.IOException

Throws:

java.io.IOException

setupOutputPath

public static org.apache.hadoop.fs.Path setupOutputPath(java.lang.String outStr,
                                                        org.apache.hadoop.fs.Path basePath,
                                                        org.apache.hadoop.mapred.JobConf conf)
                                                 throws java.io.IOException

Convenience method for setting a job's output path. If the output string is a relative path and basePath is specified, it will set the output to be a file relative to basePath. Additionally, it will remove the output if it already exists.

Parameters:

outStr - the output string as specified by the user

basePath - a base folder to be used in case outStr is a relative path

conf - the job configuration

Returns:

the final output path

Throws:

java.io.IOException

getMapredInputFormat

public static <K,V> java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>> getMapredInputFormat(org.apache.hadoop.mapred.JobConf conf,
                                                                                                              java.lang.Class<?> rawClass)

Checks if the given class is an org.apache.hadoop.mapred.InputFormat or an org.apache.hadoop.mapreduce.lib.input.FileInputFormat and returns the appropriate adapter if needed.

Parameters:

conf - the job configuration

rawClass - a class to be used as input format

Returns:

an input format class (instance of org.apache.hadoop.mapred.InputFormat)

setupOutputPath

public static org.apache.hadoop.fs.Path setupOutputPath(java.lang.String outStr,
                                                        org.apache.hadoop.fs.Path basePath,
                                                        org.apache.hadoop.mapreduce.Job job)
                                                 throws java.io.IOException

Throws:

java.io.IOException

jobCleanup

public static void jobCleanup(org.apache.hadoop.conf.Configuration conf)

Performs cleaning tasks once a job has finished.

Parameters:

conf - the job configuration

validateMapRedInputDataSet

public static void validateMapRedInputDataSet(InputDataSet inputDs,
                                              org.apache.hadoop.mapred.JobConf jobConf)
                                       throws java.lang.Exception

Throws:

java.lang.Exception

validateInputDataSet

public static void validateInputDataSet(InputDataSet inputDs,
                                        org.apache.hadoop.conf.Configuration conf)
                                 throws java.lang.Exception

Throws:

java.lang.Exception