oracle.hadoop.balancer

Class Balancer

java.lang.Object

oracle.hadoop.balancer.Balancer

public class Balancer
extends java.lang.Object

This class provides the programmatic API to run Balancer on mapred- and mapreduce-API Hadoop jobs

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	addBalancingPlan(org.apache.hadoop.conf.Configuration clientConf) Augments the supplied Configuration with information required to use the partition plan.
static org.apache.hadoop.conf.Configuration	configureCountingReducer(org.apache.hadoop.conf.Configuration conf) Configures a job for reducer load analysis.
static Balancer	createBalancer(org.apache.hadoop.conf.Configuration conf) Returns a Balancer after validating configuration properties.
static boolean	save(org.apache.hadoop.mapreduce.Job clientJob) This method is used in API mode when the client job is a mapreduce job.
static boolean	save(org.apache.hadoop.mapred.RunningJob clientJob, org.apache.hadoop.conf.Configuration conf) This method is used in API mode when the client job is a mapred job.
void	waitForCompletion() Creates a balancing plan for the job.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

addBalancingPlan

public org.apache.hadoop.conf.Configuration addBalancingPlan(org.apache.hadoop.conf.Configuration clientConf)
                                                      throws java.io.IOException

Augments the supplied Configuration with information required to use the partition plan. This method:

• Adds the partition report file name to Hadoop's distributed cache
• Sets oracle.hadoop.balancer.reportPath for clientConf
• Saves the clientConf's partitioner class name in the Configuration
• Sets the partitioner class to oracle.hadoop.balancer.mapred.BalancedPartitioner or oracle.hadoop.balancer.mapreduce.BalancedPartitioner

If any exception is thrown by this method, the supplied client configuration will not be modified to add balancing.

Parameters:

clientConf - the Configuration of the user/client job

Returns:

the modified clientConf

Throws:

java.io.IOException

configureCountingReducer

public static org.apache.hadoop.conf.Configuration configureCountingReducer(org.apache.hadoop.conf.Configuration conf)

Configures a job for reducer load analysis. This method modifies the input configuration to use a counting reducer class. Callers should set the reducer class for the job before calling this method.

Parameters:

conf - the job configuration

Returns:

the modified configuration

createBalancer

public static Balancer createBalancer(org.apache.hadoop.conf.Configuration conf)
                               throws java.io.IOException,
                                      java.lang.InterruptedException

Returns a Balancer after validating configuration properties. The Balancer class handles client jobs written in either the mapred API or the mapreduce API transparently with this single method: the parameter can either be mapred.JobConf or conf.Configuration (mapreduce)

Throws:

java.io.IOException

java.lang.InterruptedException

save

public static boolean save(org.apache.hadoop.mapreduce.Job clientJob)

This method is used in API mode when the client job is a mapreduce job. The client must call this method after the client job completes to save the job id into the output directory and save the partition report and a job analyzer report into the report directory. If Perfect Balance does not run, a partition report will not be produced.

If the client job runs successfully, the partition report will be saved in ${mapred.output.dir}/_balancer; otherwise, the partition report will be ${oracle.hadoop.balancer.reportPath}.

If configureCountingReducer(org.apache.hadoop.conf.Configuration) has been called, the job analyzer report will include detailed reducer statistics if they can be obtained from the clientJob.

The report directory will be created if it does not exist.

This method is for jobs using the org.apache.hadoop.mapreduce interface.

If an error occurs, this method prints a warning message and returns false.

Parameters:

clientJob - the submitted RunningJob

Returns:

true if reports were saved successfully

See Also:

configureCountingReducer(org.apache.hadoop.conf.Configuration)

save

public static boolean save(org.apache.hadoop.mapred.RunningJob clientJob,
           org.apache.hadoop.conf.Configuration conf)

This method is used in API mode when the client job is a mapred job. The client must call this method after the client job completes to save the job id into the output directory and save the partition report and a job analyzer report into the report directory. If Perfect Balance does not run, a partition report will not be produced.

If the client job runs successfully, the partition report will be saved in ${mapred.output.dir}/_balancer; otherwise, the partition report will be ${oracle.hadoop.balancer.reportPath}.

If configureCountingReducer(org.apache.hadoop.conf.Configuration) has been called, the job analyzer report will include detailed reducer statistics if they can be obtained from the clientJob.

The report directory will be created if it does not exist.

This method is for jobs using the org.apache.hadoop.mapred interface.

If an error occurs, this method prints a warning message and returns false.

Parameters:

clientJob - the submitted RunningJob

conf - The Configuration object of clientJob It should be clientJob.getConfiguration()

Returns:

true if reports were saved successfully

See Also:

configureCountingReducer(org.apache.hadoop.conf.Configuration)

waitForCompletion

public void waitForCompletion()
                       throws java.io.IOException,
                              java.lang.InterruptedException,
                              java.lang.ClassNotFoundException

Creates a balancing plan for the job.

Throws:

java.io.IOException

java.lang.InterruptedException

java.lang.ClassNotFoundException