Skip navigation links

oracle.hadoop.balancer
Class Balancer

java.lang.Object
oracle.hadoop.balancer.Balancer

public class Balancer

extends java.lang.Object

This class provides the programmatic-API to run Balancer on mapred- and mapreduce-API Hadoop jobs

Method Summary

org.apache.hadoop.conf.Configuration	addBalancingPlan(org.apache.hadoop.conf.Configuration clientConf) Augments the supplied Configuration with information required to use the partition plan.
static org.apache.hadoop.conf.Configuration	configureCountingReducer(org.apache.hadoop.conf.Configuration conf) Configure a job for reducer load analysis.
static Balancer	createBalancer(org.apache.hadoop.conf.Configuration conf) Returns a Balancer after validating configuration properties.
boolean	isSamplingComplete() This is a non-blocking call.
static boolean	save(org.apache.hadoop.mapreduce.Job clientJob) The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory.
static boolean	save(org.apache.hadoop.mapred.RunningJob clientJob, org.apache.hadoop.conf.Configuration conf) The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory.
void	submit() This method executes sampling.
void	waitForCompletion() Calls submit() if it has not been called.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createBalancer

public static Balancer createBalancer(org.apache.hadoop.conf.Configuration conf)
                               throws java.io.IOException,
                                      java.lang.InterruptedException

Returns a Balancer after validating configuration properties. The Balancer class handles client jobs written in either the mapred API or the mapreduce API transparently with this single method: the parameter can either be mapred.JobConf or conf.Configuration (mapreduce)

Throws:

java.io.IOException

java.lang.InterruptedException

configureCountingReducer

public static org.apache.hadoop.conf.Configuration configureCountingReducer(org.apache.hadoop.conf.Configuration conf)

Configure a job for reducer load analysis. This method modifies the input configuration to use a counting reducer class. Callers should set the reducer class for the job before calling this method. The configuration property "oracle.hadoop.balancer.tools.useCountingReducer" is set to true.

Parameters:

conf - the job configuration

Returns:

the modified configuration

addBalancingPlan

public org.apache.hadoop.conf.Configuration addBalancingPlan(org.apache.hadoop.conf.Configuration clientConf)
                                                      throws java.io.IOException

Augments the supplied Configuration with information required to use the partition plan. This method:

• Saves the clientConf's partitioner class name in the the Configuration
• Sets the partitioner class to oracle.hadoop.balancer.mapred.BalancedPartitioner or oracle.hadoop.balancer.mapreduce.BalancedPartitioner
• Sets oracle.hadoop.balancer.reportPath for clientConf
• Adds the partition report file name to Hadoop's distributed cache

Parameters:

clientConf - the Configuration of the user/client job

Returns:

the modified clientConf

Throws:

java.io.IOException

submit

public void submit()
            throws java.io.IOException,
                   java.lang.InterruptedException,
                   java.lang.ClassNotFoundException

This method executes sampling. Subsequent calls to this method do nothing (same behavior as Hadoop's @code{Job::submit()})

Throws:

java.io.IOException

java.lang.InterruptedException

java.lang.ClassNotFoundException

waitForCompletion

public void waitForCompletion()
                       throws java.io.IOException,
                              java.lang.InterruptedException,
                              java.lang.ClassNotFoundException

Calls submit() if it has not been called. Waits for the sampling started by submit() to complete before returning.

Throws:

java.io.IOException

java.lang.InterruptedException

java.lang.ClassNotFoundException

isSamplingComplete

public boolean isSamplingComplete()

This is a non-blocking call.

Returns:

true if the sampling started by submit() has completed.

save

public static boolean save(org.apache.hadoop.mapreduce.Job clientJob)

The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory. This report will include reducer statistics if they can be obtained from the clientJob. The report directory will be created if it does not exist.

If the client job runs successfully, the partition report will be saved in ${mapred.output.dir}/_balancer; otherwise, the partition report will be ${oracle.hadoop.balancer.reportPath}.

This method is for jobs using the org.apache.hadoop.mapreduce interface.

If an error occurs, this method prints a warning message and returns false.

Parameters:

clientJob -

Returns:

true if all balancer state was saved successfully

save

public static boolean save(org.apache.hadoop.mapred.RunningJob clientJob,
                           org.apache.hadoop.conf.Configuration conf)

The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory. This report will include reducer statistics if they can be obtained from the clientJob. The report directory will be created if it does not exist.

If the client job runs successfully, the partition report will be saved in ${mapred.output.dir}/_balancer; otherwise, the partition report will be ${oracle.hadoop.balancer.reportPath}.

This method is for jobs using the org.apache.hadoop.mapred interface.

If an error occurs, this method prints a warning message and returns false.

Parameters:

clientJob - the submitted RunningJob

conf - The Configuration object of clientJob It should be clientJob.getConfiguration()

Returns:

true if all balancer state was saved successfully

Skip navigation links