Debugging odcp
You must configure the cluster to enable debugging for odcp.
To configure the cluster:
-
As the
root
user, add following lines to/etc/hadoop/conf/log4j.properties
on each node of the cluster:log4j.logger.oracle.paas.bdcs.conductor=DEBUG log4j.logger.org.apche.hadoop.fs.swift.http=DEBUG
Or, to configure all nodes:
$ dcli -c $NODES "echo 'log4j.logger.oracle.paas.bdcs.conductor=DEBUG' >> /etc/hadoop/conf/log4j.properties" $ dcli -c $NODES "echo 'log4j.logger.org.apche.hadoop.fs.swift.http=DEBUG' >> /etc/hadoop/conf/log4j.properties"
-
As the
oracle
user, find the logs in following HDFS directory:hdfs:///tmp/logs/username/logs/application_application_ID/
For example:
$ hadoop fs -ls /tmp/logs/oracle/logs/ Found 15 items drwxrwx--- - oracle hadoop 0 2016-08-23 07:29 /tmp/logs/oracle/logs/application_14789235086687_0029 drwxrwx--- - oracle hadoop 0 2016-08-23 08:07 /tmp/logs/oracle/logs/application_14789235086687_0030 drwxrwx--- - oracle hadoop 0 2016-08-23 08:20 /tmp/logs/oracle/logs/application_14789235086687_0001 drwxrwx--- - oracle hadoop 0 2016-08-23 10:19 /tmp/logs/oracle/logs/application_14789235086687_0002 drwxrwx--- - oracle hadoop 0 2016-08-23 10:20 /tmp/logs/oracle/logs/application_14789235086687_0003 drwxrwx--- - oracle hadoop 0 2016-08-23 10:40 /tmp/logs/oracle/logs/application_14789235086687_0004 ... # cat logs as: hadoop fs -cat /tmp/logs/oracle/logs/application_1469028504906_0032/slclbv0036.em3.oraclecloud.com_8041 # copy to local FShadoop fs -copyToLocal /tmp/logs/oracle/logs/application_1469028504906_0032/slclbv0036.em3.oraclecloud.com_8041 /tmp/log/slclbv0036.em3.oraclecloud.com_8041
Collecting Transfer Rates
You can collect the transfer rates when debugging is enabled. Transfer rates are reported after every:
-
Read chunk operation
-
Write/upload chunk operation
The summary throughput is reported after a chunk transfer is completed. The summary throughput includeds all:
-
Read operations
-
Write/upload operations
-
Spark framework operations (task distribution, task management, etc.)
Output Example:
./get-transfer-rates.sh application_1476272395108_0054 2>/dev/null
Action,Speed [MBps],Start time,End time,Duration [s],Size [B]
Download from OSS,2.5855451864420473,2016-10-31 11:34:48,2016-10-31 11:38:06,198.024,536870912
Download from OSS,2.548912231791706,2016-10-31 11:34:47,2016-10-31 11:38:08,200.87,536870912
Download from OSS,2.53447780846872,2016-10-31 11:34:47,2016-10-31 11:38:09,202.014,536870912
Download from OSS,2.5130931169717226,2016-10-31 11:34:48,2016-10-31 11:38:11,203.733,536870912
Write to HDFS,208.04550995530275,2016-10-31 14:00:30,2016-10-31 14:00:33,2.46099999999999967435,536870912
Write to HDFS,271.76220806794055,2016-10-31 14:00:38,2016-10-31 14:00:40,1.88400000000000001398,536870912
Write to HDFS,277.5067750677507,2016-10-31 14:00:43,2016-10-31 14:00:45,1.84499999999999985045,536870912
Write to HDFS,218.0579216354344,2016-10-31 14:00:44,2016-10-31 14:00:46,2.34800000000000013207,536870912
Write to HDFS,195.56913674560735,2016-10-31 14:00:44,2016-10-31 14:00:47,2.61799999999999978370,536870912
Use the following command to collect output rates:
get-transfer-rates.sh application_ID