Debugging odcp

You must configure the cluster to enable debugging for odcp.

To configure the cluster:

As the root user, add following lines to /etc/hadoop/conf/log4j.properties on each node of the cluster:

log4j.logger.oracle.paas.bdcs.conductor=DEBUG
log4j.logger.org.apche.hadoop.fs.swift.http=DEBUG

Or, to configure all nodes:

$ dcli -c $NODES "echo 'log4j.logger.oracle.paas.bdcs.conductor=DEBUG' >> /etc/hadoop/conf/log4j.properties"
$ dcli -c $NODES "echo 'log4j.logger.org.apche.hadoop.fs.swift.http=DEBUG' >> /etc/hadoop/conf/log4j.properties"

As the oracle user, find the logs in following HDFS directory:

hdfs:///tmp/logs/username/logs/application_application_ID/

For example:

$ hadoop fs -ls /tmp/logs/oracle/logs/
Found 15 items
drwxrwx---   - oracle hadoop       0 2016-08-23 07:29 /tmp/logs/oracle/logs/application_14789235086687_0029
drwxrwx---   - oracle hadoop       0 2016-08-23 08:07 /tmp/logs/oracle/logs/application_14789235086687_0030
drwxrwx---   - oracle hadoop       0 2016-08-23 08:20 /tmp/logs/oracle/logs/application_14789235086687_0001
drwxrwx---   - oracle hadoop       0 2016-08-23 10:19 /tmp/logs/oracle/logs/application_14789235086687_0002
drwxrwx---   - oracle hadoop       0 2016-08-23 10:20 /tmp/logs/oracle/logs/application_14789235086687_0003
drwxrwx---   - oracle hadoop       0 2016-08-23 10:40 /tmp/logs/oracle/logs/application_14789235086687_0004
...

# cat logs as:
hadoop fs -cat /tmp/logs/oracle/logs/application_1469028504906_0032/slclbv0036.em3.oraclecloud.com_8041

# copy to local FShadoop fs -copyToLocal /tmp/logs/oracle/logs/application_1469028504906_0032/slclbv0036.em3.oraclecloud.com_8041 /tmp/log/slclbv0036.em3.oraclecloud.com_8041

Collecting Transfer Rates

You can collect the transfer rates when debugging is enabled. Transfer rates are reported after every:

Read chunk operation
Write/upload chunk operation

The summary throughput is reported after a chunk transfer is completed. The summary throughput includeds all:

Read operations
Write/upload operations
Spark framework operations (task distribution, task management, etc.)

Output Example:

./get-transfer-rates.sh application_1476272395108_0054 2>/dev/null
Action,Speed [MBps],Start time,End time,Duration [s],Size [B]
Download from OSS,2.5855451864420473,2016-10-31 11:34:48,2016-10-31 11:38:06,198.024,536870912
Download from OSS,2.548912231791706,2016-10-31 11:34:47,2016-10-31 11:38:08,200.87,536870912
Download from OSS,2.53447780846872,2016-10-31 11:34:47,2016-10-31 11:38:09,202.014,536870912
Download from OSS,2.5130931169717226,2016-10-31 11:34:48,2016-10-31 11:38:11,203.733,536870912
Write to HDFS,208.04550995530275,2016-10-31 14:00:30,2016-10-31 14:00:33,2.46099999999999967435,536870912
Write to HDFS,271.76220806794055,2016-10-31 14:00:38,2016-10-31 14:00:40,1.88400000000000001398,536870912
Write to HDFS,277.5067750677507,2016-10-31 14:00:43,2016-10-31 14:00:45,1.84499999999999985045,536870912
Write to HDFS,218.0579216354344,2016-10-31 14:00:44,2016-10-31 14:00:46,2.34800000000000013207,536870912
Write to HDFS,195.56913674560735,2016-10-31 14:00:44,2016-10-31 14:00:47,2.61799999999999978370,536870912

Use the following command to collect output rates:

get-transfer-rates.sh application_ID