oracle.spatial.hadoop.vector.mapred.input

Class CompositeInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<K,V>

oracle.spatial.hadoop.vector.mapred.input.CompositeInputFormat<K,V>

Type Parameters:

K - The type of the record's key. The same used by internalInputFormat

V - The type of the record's value. The same used by internalInputFormat.

All Implemented Interfaces:

AbstractCompositeInputFormat<K,V>, org.apache.hadoop.mapred.InputFormat<K,V>, org.apache.hadoop.mapred.JobConfigurable

Direct Known Subclasses:

FileSplitInputFormat, SpatialFilterInputFormat

public abstract class CompositeInputFormat<K,V>
extends org.apache.hadoop.mapred.FileInputFormat<K,V>
implements org.apache.hadoop.mapred.JobConfigurable, AbstractCompositeInputFormat<K,V>

Abstract InputFormat which uses another InputFormat (internalInputFormat) to read data and allows adding additional functionality such as filtering the data. The internal input format must be a subclass of FileInputFormat or CombineFileInputFormat.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapred.InputFormat<K,V>	iInputFormat

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

LOG

Constructor Summary

Constructors

Constructor and Description
CompositeInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(org.apache.hadoop.mapred.JobConf conf)
abstract org.apache.hadoop.mapred.RecordReader<K,V>	getDelegateRecordReader(org.apache.hadoop.mapred.FileSplit split, org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.mapred.Reporter reporter) Creates a record reader instance.
static org.apache.hadoop.mapred.InputSplit	getFittingInputSplit(org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.mapred.InputFormat<?,?> inputFormat, org.apache.hadoop.mapred.InputSplit split) Returns an instance of an InputSplit subclass appropriate for the given InputFormat.
org.apache.hadoop.mapred.InputFormat<K,V>	getInternalInputFormat(org.apache.hadoop.mapred.JobConf jobConf) Gets an instance of the class set as the internal input format
static <K,V> java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>>	getInternalInputFormatClass(org.apache.hadoop.mapred.JobConf jobConf) Gets the internal input format type which is used to actually read the data.
static <K,V> RecordInfoProvider<K,V>	getRecordInfoProvider(RecordInfoProvider<K,V> provider, org.apache.hadoop.mapred.JobConf conf) Gets an instance of the specified RecordInfoProvider implementation
static <K,V> java.lang.Class<? extends RecordInfoProvider<K,V>>	getRecordInfoProviderClass(org.apache.hadoop.mapred.JobConf jobConf) Gets the class of the RecordInfoProvider implementation
org.apache.hadoop.mapred.RecordReader<K,V>	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.mapred.Reporter reporter)
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf conf, int splitCount)
static <K,V> void	setInternalInputFormatClass(org.apache.hadoop.mapred.JobConf jobConf, java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>> iInputFormat) Sets the internal input format type which is used to actually read the data.
static <K,V> void	setRecordInfoProviderClass(org.apache.hadoop.mapred.JobConf jobConf, java.lang.Class<? extends RecordInfoProvider<K,V>> provider) Sets the class of the RecordInfoProvider implementation

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, isSplitable, listStatus, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

iInputFormat

protected org.apache.hadoop.mapred.InputFormat<K,V> iInputFormat

Constructor Detail

CompositeInputFormat

public CompositeInputFormat()

Method Detail

setInternalInputFormatClass

public static <K,V> void setInternalInputFormatClass(org.apache.hadoop.mapred.JobConf jobConf,
                                                     java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>> iInputFormat)

Sets the internal input format type which is used to actually read the data.

Parameters:

jobConf - the job configuration

iInputFormat - a class which is a subclass of FileInputFormat or CombineFileInputFormat

getInternalInputFormatClass

public static <K,V> java.lang.Class<? extends org.apache.hadoop.mapred.InputFormat<K,V>> getInternalInputFormatClass(org.apache.hadoop.mapred.JobConf jobConf)

Gets the internal input format type which is used to actually read the data.

Parameters:

jobConf - the job configuration

Returns:

a class which is a subclass of FileInputFormat or CombineFileInputFormat

getInternalInputFormat

public org.apache.hadoop.mapred.InputFormat<K,V> getInternalInputFormat(org.apache.hadoop.mapred.JobConf jobConf)

Gets an instance of the class set as the internal input format

Specified by:

getInternalInputFormat in interface AbstractCompositeInputFormat<K,V>

Parameters:

jobConf - the job configuration

Returns:

an InputFormat instance

setRecordInfoProviderClass

public static <K,V> void setRecordInfoProviderClass(org.apache.hadoop.mapred.JobConf jobConf,
                                                    java.lang.Class<? extends RecordInfoProvider<K,V>> provider)

Sets the class of the RecordInfoProvider implementation

Parameters:

jobConf - the job configuration

provider - a class that extends from RecordInfoProvider

getRecordInfoProviderClass

public static <K,V> java.lang.Class<? extends RecordInfoProvider<K,V>> getRecordInfoProviderClass(org.apache.hadoop.mapred.JobConf jobConf)

Gets the class of the RecordInfoProvider implementation

Parameters:

jobConf - the job configuration

Returns:

a class that extends from RecordInfoProvider

getRecordInfoProvider

public static <K,V> RecordInfoProvider<K,V> getRecordInfoProvider(RecordInfoProvider<K,V> provider,
                                                                  org.apache.hadoop.mapred.JobConf conf)

Gets an instance of the specified RecordInfoProvider implementation

Parameters:

provider - if null, a new instance will be returned, otherwise, the same instance will be returned. It is used to release the caller from checking if the instance is null.

conf - the job configuration

Returns:

a RecordInfoProvider instance

configure

public void configure(org.apache.hadoop.mapred.JobConf conf)

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf conf,
                                                       int splitCount)
                                                throws java.io.IOException

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<K,V>

Overrides:

getSplits in class org.apache.hadoop.mapred.FileInputFormat<K,V>

Throws:

java.io.IOException

getRecordReader

public org.apache.hadoop.mapred.RecordReader<K,V> getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                                  org.apache.hadoop.mapred.JobConf conf,
                                                                  org.apache.hadoop.mapred.Reporter reporter)
                                                           throws java.io.IOException

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<K,V>

Specified by:

getRecordReader in class org.apache.hadoop.mapred.FileInputFormat<K,V>

Throws:

java.io.IOException

getDelegateRecordReader

public abstract org.apache.hadoop.mapred.RecordReader<K,V> getDelegateRecordReader(org.apache.hadoop.mapred.FileSplit split,
                                                                                   org.apache.hadoop.mapred.JobConf conf,
                                                                                   org.apache.hadoop.mapred.Reporter reporter)
                                                                            throws java.io.IOException

Creates a record reader instance. The returned record instance can be a wrapper of the internalInputFormat's record reader. This method is called after getRecordReader(). If the internalInputFormat is a CombineFileInputFormat subclass, this method can be called as many times as paths are contained in the CombineFileSplits.

Parameters:

split - a FileSplit instance

conf - the job configuration

reporter - a Reporter instance

Returns:

a RecordReader instance.

Throws:

java.io.IOException

getFittingInputSplit

public static org.apache.hadoop.mapred.InputSplit getFittingInputSplit(org.apache.hadoop.mapred.JobConf conf,
                                                                       org.apache.hadoop.mapred.InputFormat<?,?> inputFormat,
                                                                       org.apache.hadoop.mapred.InputSplit split)
                                                                throws java.io.IOException

Returns an instance of an InputSplit subclass appropriate for the given InputFormat. For example, if the passed split is a FileSplit instance and the InputFormat is a CombineFileInputFormat implementation, the returned split will be a instance of CombineFileSplit.

Parameters:

conf - the job configuration

inputFormat - an InputFormat instance

split - an InputSplit instance

Returns:

if the passed split fits the input format, it is returned, otherwise a new one that fits is returned.

Throws:

java.io.IOException