Skip navigation links

oracle.hadoop.balancer
Class Balancer

java.lang.Object
oracle.hadoop.balancer.Balancer

public class Balancer

extends java.lang.Object

This class provides the programmatic-API to run Balancer on mapred- and mapreduce-API Hadoop jobs

Method Summary

org.apache.hadoop.conf.Configuration	addBalancingPlan(org.apache.hadoop.conf.Configuration clientConf) Augments the supplied Configuration with information required to use the partition plan.
static org.apache.hadoop.conf.Configuration	configureCountingReducer(org.apache.hadoop.conf.Configuration conf) Configures a job for reducer load analysis.
static Balancer	createBalancer(org.apache.hadoop.conf.Configuration conf) Returns a Balancer after validating configuration properties.
static boolean	save(org.apache.hadoop.mapreduce.Job clientJob) The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory.
static boolean	save(org.apache.hadoop.mapred.RunningJob clientJob, org.apache.hadoop.conf.Configuration conf) The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory.
void	waitForCompletion() Creates a balancing plan for the job.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

addBalancingPlan

public org.apache.hadoop.conf.Configuration addBalancingPlan(org.apache.hadoop.conf.Configuration clientConf)
                                                      throws java.io.IOException

Augments the supplied Configuration with information required to use the partition plan. This method:

• Adds the partition report file name to Hadoop's distributed cache
• Sets oracle.hadoop.balancer.reportPath for clientConf
• Saves the clientConf's partitioner class name in the the Configuration
• Sets the partitioner class to oracle.hadoop.balancer.mapred.BalancedPartitioner or oracle.hadoop.balancer.mapreduce.BalancedPartitioner

If any exception is thrown by this method, the supplied client configuration will not be modified to add balancing.

Parameters:

clientConf - the Configuration of the user/client job

Returns:

the modified clientConf

Throws:

java.io.IOException

configureCountingReducer

public static org.apache.hadoop.conf.Configuration configureCountingReducer(org.apache.hadoop.conf.Configuration conf)

Configures a job for reducer load analysis. This method modifies the input configuration to use a counting reducer class. Callers should set the reducer class for the job before calling this method.

Parameters:

conf - the job configuration

Returns:

the modified configuration

createBalancer

public static Balancer createBalancer(org.apache.hadoop.conf.Configuration conf)
                               throws java.io.IOException,
                                      java.lang.InterruptedException

Returns a Balancer after validating configuration properties. The Balancer class handles client jobs written in either the mapred API or the mapreduce API transparently with this single method: the parameter can either be mapred.JobConf or conf.Configuration (mapreduce)

Throws:

java.io.IOException

java.lang.InterruptedException

save

public static boolean save(org.apache.hadoop.mapreduce.Job clientJob)

The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory. This report will include reducer statistics if they can be obtained from the clientJob. The report directory will be created if it does not exist.

If the client job runs successfully, the partition report will be saved in ${mapred.output.dir}/_balancer; otherwise, the partition report will be ${oracle.hadoop.balancer.reportPath}.

This method is for jobs using the org.apache.hadoop.mapreduce interface.

If an error occurs, this method prints a warning message and returns false.

Parameters:

clientJob -

Returns:

true if all balancer state was saved successfully

save

public static boolean save(org.apache.hadoop.mapred.RunningJob clientJob,
                           org.apache.hadoop.conf.Configuration conf)

The client must call this method after the client job completes to save the partition report and a job analyzer report into the report directory. This report will include reducer statistics if they can be obtained from the clientJob. The report directory will be created if it does not exist.

If the client job runs successfully, the partition report will be saved in ${mapred.output.dir}/_balancer; otherwise, the partition report will be ${oracle.hadoop.balancer.reportPath}.

This method is for jobs using the org.apache.hadoop.mapred interface.

If an error occurs, this method prints a warning message and returns false.

Parameters:

clientJob - the submitted RunningJob

conf - The Configuration object of clientJob It should be clientJob.getConfiguration()

Returns:

true if all balancer state was saved successfully

waitForCompletion

public void waitForCompletion()
                       throws java.io.IOException,
                              java.lang.InterruptedException,
                              java.lang.ClassNotFoundException

Creates a balancing plan for the job.

Throws:

java.io.IOException

java.lang.InterruptedException

java.lang.ClassNotFoundException

Skip navigation links