Package oracle.kv.hadoop.table

Class TableInputFormatBase<K,V>

java.lang.Object
org.apache.hadoop.mapreduce.InputFormat<K,V>
oracle.kv.hadoop.table.TableInputFormatBase<K,V>
Direct Known Subclasses:
TableInputFormat
public abstract class TableInputFormatBase<K,V> extends InputFormat<K,V>
This is the base class for Oracle NoSQL Database InputFormat classes that can be used to run MapReduce against data stored via the Table API. Keys are of type PrimaryKey. Values are always of type Row.

Parameters may be passed using either the static setters on this class or through the Hadoop JobContext configuration parameters. The following parameters are recognized:

  • oracle.kv.kvstore - the KV Store name for this InputFormat to operate on. This is equivalent to the setKVStoreName(java.lang.String) method.
  • oracle.kv.hosts - one or more hostname:port pairs separated by commas naming hosts in the KV Store. This is equivalent to the setKVHelperHosts(java.lang.String[]) method.
  • oracle.kv.hadoop.hosts - one or more hostname strings separated by commas naming the Hadoop data node hosts in the Hadoop cluster that will support MapReduce jobs. This is equivalent to the setKVHadoopHosts(java.lang.String[]) method. The value(s) specified by this property will be returned by the getLocations method of TableInputSplit. If this property is not specified, or if the setKVHadoopHosts(java.lang.String[]) method is not called, then the values specified via the oracle.kv.hosts property (or the setKVHelperHosts(java.lang.String[]) method) will be used instead.
  • oracle.kv.tableName - the name of the table in the store from which data will be retrieved. This is equivalent to the setTableName(java.lang.String) method.
  • oracle.kv.primaryKey - Property whose value consists of the components to use when constructing the key to employ when iterating the table. The format of this property's value must be a list of name:value pairs in JSON FORMAT like the following: -Doracle.kv.primaryKey="{\"name\":\"stringVal\",\"name\":floatVal}" where the list itself is enclosed in un-escaped double quotes and corresponding curly brace; and each field name component -- as well as each STRING type field value component -- is enclosed in ESCAPED double quotes.

    In addition to the JSON format requirement above, the values referenced by the various fieldValue components of this Property must satisfy the semantics of PrimaryKey for the given table; that is, they must represent a first-to-last subset of the table's primary key fields, and they must be specified in the same order as those primary key fields. If the components of this property do not satisfy these requirements, a full primary key wildcard will be used when iterating the table.

    This is equivalent to the setPrimaryKeyProperty(java.lang.String) method.

  • oracle.kv.fieldRange - Property whose value consists of the components to use when constructing the field range to employ when iterating the table. The format of this property's value must be a list of name:value pairs in JSON FORMAT like the following: -Doracle.kv.fieldRange="{\"name\":\"fieldName\", \"start\":\"startVal\",[\"startInclusive\":true|false], \"end\"\"endVal\",[\"endInclusive\":true|false]}" where for the given field over which to range, the 'start', and 'end' components are required, and the 'startInclusive' and 'endInclusive' components are optional; defaulting to 'true' if not included. Note that the list itself is enclosed in un-escaped double quotes and corresponding curly brace; and each name component and string type value component is enclosed in ESCAPED double quotes.

    In addition to the JSON format requirement above, the values referenced by the components of this Property's value must also satisfy the semantics of FieldRange; that is,

    • the values associated with the target key must correspond to a valid primary key in the table
    • the value associated with the fieldName must be the name of a valid field of the primary key over which iteration will be performed
    • the values associated with the start and end of the range must correspond to valid values of the given fieldName
    • the value associated with either of the inclusive components must be either 'true' or 'false'
    If the components of this property do not satisfy these requirements, then table iteration will be performed over the full range of values of the PrimaryKey iteration rather than a sub-range.

    This is equivalent to the setFieldRangeProperty(java.lang.String) method.

  • oracle.kv.consistency - Specifies the read consistency associated with the lookup of the child KV pairs. Version- and Time-based consistency may not be used. If null, the default consistency is used.

    This is equivalent to the setConsistency(oracle.kv.Consistency) method.

  • oracle.kv.timeout - Specifies an upper bound on the time interval for processing a particular KV retrieval. A best effort is made to not exceed the specified limit. If zero, the default request timeout is used. This value is always in milliseconds.

    This is equivalent to the setTimeout(long) and setTimeoutUnit(java.util.concurrent.TimeUnit) methods.

  • oracle.kv.maxRequests - Specifies the maximum number of client side threads to use when running an iteration; where a value of 1 causes the iteration to be performed using only the current thread, and a value of 0 causes the client to base the number of threads to employ on the current store topology.

    This is equivalent to the setMaxRequests(int) method.

  • oracle.kv.batchSize - Specifies the suggested number of keys to fetch during each network round trip by the InputFormat. If 0, an internally determined default is used. This is equivalent to the setBatchSize(int) method.
  • oracle.kv.maxBatches - Specifies the maximum number of result batches that can be held in memory on the client side before processing on the server side pauses. This parameter can be used to prevent the client side memory from being exceeded if the client cannot consume results as fast as they are generated by the server side.

    This is equivalent to the setMaxBatches(int) method.

Internally, the TableInputFormatBase class utilizes the method oracle.kv.table.TableIterator<oracle.kv.table.Row> TableAPI.tableIterator to retrieve records. You should refer to the javadoc for that method for information about the various parameters.

TableInputFormatBase dynamically generates a number of splits, each encapsulating a list of sets in which the elements of each set are partition ids over which can be retrieved in parallel; to optimize retrieval performance. The "size" of each split that is generated -- which will be the value returned by the getLength method of TableInputSplit -- is the number of that encapsulated list of partition id sets. If the consistency passed to TableInputFormatBase is NONE_REQUIRED (the default), then InputSplit.getLocations() will return an array of the names of the master and the replica(s) which contain the partition. Alternatively, if the consistency is ABSOLUTE, then the array returned will contain only the name of the master. This means that if Hadoop job trackers are running on the nodes named in the returned location array, Hadoop will generally attempt to run the subtasks for a particular partition on those nodes where the data is stored and replicated. Hadoop and Oracle NoSQL DB administrators should be careful about co-location of Oracle NoSQL DB and Hadoop processes since they may compete for resources.

InputSplit.getLength() always returns 1.

A simple example demonstrating the Oracle NoSQL DB Hadoop oracle.kv.hadoop.table.TableInputFormat class reading data from Hadoop in a MapReduce job and counting the number of rows in a given table in the store can be found in the KVHOME/examples/hadoop/table directory. The javadoc for that program describes the simple MapReduce processing as well as how to invoke the program in Hadoop.

Since:
3.1

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    TableInputFormatBase.TopologyLocatorWrapper
    Special class that wraps the static get method of the TopologyLocator utility.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected TableInputFormatBase.TopologyLocatorWrapper
    topologyLocator
     
    protected static final String
    USER_SECURITY_DIR
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static void
    clearLocalKVSecurity()
    Convenience method for testing.
    static void
    setBatchSize(int batchSize)
    Specifies the suggested number of keys to fetch during each network round trip by the InputFormat.
    static void
    setConsistency(Consistency consistency)
    Specifies the read consistency associated with the lookup of the child KV pairs.
    static void
    setDirection(Direction newDirection)
    Specifies the order in which records are returned by the InputFormat.
    static void
    setFieldRangeProperty(String newProperty)
    Sets the String to use for the property value whose contents are used to construct the field range to employ when iterating the table.
    static void
    setKVHadoopHosts(String[] newHadoopHosts)
    Set the KV Hadoop data node host name(s) for this InputFormat to operate on.
    static void
    setKVHelperHosts(String[] newHelperHosts)
    Set the KV Helper host:port pair(s) for this InputFormat to operate on.
    static void
    setKVSecurity(String loginFile, PasswordCredentials userPasswordCredentials, String trustFile)
    Sets the login properties file and the public trust file (keys and/or certificates), as well as the PasswordCredentials for authentication.
    static void
    setKVStoreName(String newStoreName)
    Set the KV Store name for this InputFormat to operate on.
    static void
    setMaxBatches(int newMaxBatches)
    Specifies the maximum number of result batches that can be held in memory on the client side before processing on the server side pauses.
    static void
    setMaxRequests(int newMaxRequests)
    Specifies the maximum number of client side threads to use when running an iteration; where a value of 1 causes the iteration to be performed using only the current thread, and a value of 0 causes the client to base the number of threads to employ on the current store topology.
    static void
    setPrimaryKeyProperty(String newProperty)
    Sets the String to use for the property value whose contents are used to construct the primary key to employ when iterating the table.
    void
    setQueryInfo(int newQueryBy, String newWhereClause, Integer newPartitionId)
     
    static void
    setTableName(String newTableName)
    Set the name of the table in the KV store that this InputFormat will operate on.
    static void
    setTimeout(long timeout)
    Specifies an upper bound on the time interval for processing a particular KV retrieval.
    static void
    setTimeoutUnit(TimeUnit timeoutUnit)
    Specifies the unit of the timeout parameter.

    Methods inherited from class org.apache.hadoop.mapreduce.InputFormat

    createRecordReader

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Method Details

    • setKVStoreName

      public static void setKVStoreName(String newStoreName)
      Set the KV Store name for this InputFormat to operate on. This is equivalent to passing the oracle.kv.kvstore Hadoop property.
      Parameters:
      newStoreName - the new KV Store name to set

    • setKVHelperHosts

      public static void setKVHelperHosts(String[] newHelperHosts)
      Set the KV Helper host:port pair(s) for this InputFormat to operate on. This is equivalent to passing the oracle.kv.hosts Hadoop property.
      Parameters:
      newHelperHosts - array of hostname:port strings of any hosts in the KV Store.

    • setKVHadoopHosts

      public static void setKVHadoopHosts(String[] newHadoopHosts)
      Set the KV Hadoop data node host name(s) for this InputFormat to operate on. This is equivalent to passing the oracle.kv.hadoop.hosts property.
      Parameters:
      newHadoopHosts - array of hostname strings corresponding to the names of the Hadoop data node hosts in the Hadoop cluster that this InputFormat will use to support MapReduce jobs.

    • setTableName

      public static void setTableName(String newTableName)
      Set the name of the table in the KV store that this InputFormat will operate on. This is equivalent to passing the oracle.kv.tableName property.
      Parameters:
      newTableName - the new table name to set.

    • setPrimaryKeyProperty

      public static void setPrimaryKeyProperty(String newProperty)
      Sets the String to use for the property value whose contents are used to construct the primary key to employ when iterating the table. The format of the String input to this method must be a comma-separated String of the form: fieldName,fieldValue,fieldType,fieldName,fieldValue,fieldType,.. where the number of elements separated by commas must be a multiple of 3, and each fieldType must be 'STRING', 'INTEGER', 'LONG', 'FLOAT', 'DOUBLE', or 'BOOLEAN'. Additionally, the values referenced by the various fieldType and fieldValue components of this String must satisfy the semantics of PrimaryKey for the given table; that is, they must represent a first-to-last subset of the table's primary key fields, and they must be specified in the same order as those primary key fields. If the String referenced by this property does not satisfy these requirements, a full primary key wildcard will be used when iterating the table.

      This is equivalent to passing the oracle.kv.primaryKey Hadoop property.

      Parameters:
      newProperty - the new shard key property to set

    • setFieldRangeProperty

      public static void setFieldRangeProperty(String newProperty)
      Sets the String to use for the property value whose contents are used to construct the field range to employ when iterating the table. The format of this property's value must be a list of name:value pairs in JSON FORMAT like the following: -Doracle.kv.fieldRange="{\"name\":\"fieldName\", \"start\":\"startVal\",[\"startInclusive\":true|false], \"end\"\"endVal\",[\"endInclusive\":true|false]}" where for the given field over which to range, the 'start', and 'end' components are required, and the 'startInclusive' and 'endInclusive' components are optional; defaulting to 'true' if not included. Note that the list itself is enclosed in un-escaped double quotes and corresponding curly brace; and each name component and string type value component is enclosed in ESCAPED double quotes.

      In addition to the JSON format requirement above, the values referenced by the components of this Property's value must also satisfy the semantics of FieldRange; that is,

      • the values associated with the target key must correspond to a valid primary key in the table
      • the value associated with the fieldName must be the name of a valid field of the primary key over which iteration will be performed
      • the values associated with the start and end of the range must correspond to valid values of the given fieldName
      • the value associated with either of the inclusive components must be either 'true' or 'false'
      If the components of this property do not satisfy these requirements, then table iteration will be performed over the full range of values of the PrimaryKey iteration rather than a sub-range.

      This is equivalent to passing the oracle.kv.fieldRange Hadoop property.

      Parameters:
      newProperty - the new field range property to set

    • setDirection

      public static void setDirection(Direction newDirection)
      Specifies the order in which records are returned by the InputFormat. Note that when doing PrimaryKey iteration, only Direction.UNORDERED is allowed.
      Parameters:
      newDirection - the direction to retrieve data

    • setConsistency

      public static void setConsistency(Consistency consistency)
      Specifies the read consistency associated with the lookup of the child KV pairs. Version- and Time-based consistency may not be used. If null, the default consistency is used. This is equivalent to passing the oracle.kv.consistency Hadoop property.
      Parameters:
      consistency - the consistency

    • setTimeout

      public static void setTimeout(long timeout)
      Specifies an upper bound on the time interval for processing a particular KV retrieval. A best effort is made to not exceed the specified limit. If zero, the default request timeout is used. This is equivalent to passing the oracle.kv.timeout Hadoop property.
      Parameters:
      timeout - the timeout

    • setTimeoutUnit

      public static void setTimeoutUnit(TimeUnit timeoutUnit)
      Specifies the unit of the timeout parameter. The value input may be null, only if timeout is currently zero; in which case an IllegalArgumentException is thrown. This is equivalent to passing the oracle.kv.timeout.unit Hadoop property.
      Parameters:
      timeoutUnit - the timeout unit

    • setMaxRequests

      public static void setMaxRequests(int newMaxRequests)
      Specifies the maximum number of client side threads to use when running an iteration; where a value of 1 causes the iteration to be performed using only the current thread, and a value of 0 causes the client to base the number of threads to employ on the current store topology.

      This is equivalent to passing the oracle.kv.maxRequests Hadoop property.

      Parameters:
      newMaxRequests - the suggested number of threads to employ when an iteration.

    • setBatchSize

      public static void setBatchSize(int batchSize)
      Specifies the suggested number of keys to fetch during each network round trip by the InputFormat. If 0, an internally determined default is used. This is equivalent to passing the oracle.kv.batchSize Hadoop property.
      Parameters:
      batchSize - the suggested number of keys to fetch during each network round trip.

    • setMaxBatches

      public static void setMaxBatches(int newMaxBatches)
      Specifies the maximum number of result batches that can be held in memory on the client side before processing on the server side pauses. This parameter can be used to prevent the client side memory from being exceeded if the client cannot consume results as fast as they are generated by the server side.

      This is equivalent to passing the oracle.kv.maxBatches Hadoop property.

      Parameters:
      newMaxBatches - the suggested number of threads to employ when an iteration.

    • setKVSecurity

      public static void setKVSecurity(String loginFile, PasswordCredentials userPasswordCredentials, String trustFile) throws IOException
      Sets the login properties file and the public trust file (keys and/or certificates), as well as the PasswordCredentials for authentication. The value of the loginFile and trustFile parameters must be either a fully qualified path referencing a file located on the local file system, or the name of a file (no path) whose contents can be retrieved as a resource from the current VM's classpath.

      Note that this class provides the getSplits method; which must be able to contact a secure store, and so will need access to local copies of the login properties and trust files. As a result, if the values input for the loginFile and trustFile parameters are simple file names rather than fully qualified paths, this method will retrieve the contents of each from the classpath and generate private, local copies of the associated file for availability to the getSplits method.

      Throws:
      IOException

    • setQueryInfo

      public void setQueryInfo(int newQueryBy, String newWhereClause, Integer newPartitionId)

    • clearLocalKVSecurity

      public static void clearLocalKVSecurity()
      Convenience method for testing. A test calls this method when a test run prior to the current test has set the security related fields to values that may interfere with the current test.