About MapReduce Jobs
You can submit MapReduce jobs using the cluster console, the REST API, or the command line interface.
Note:
-
The MapReduce API is based on
org.apache.hadoop.mapreduce.Job
and creates its own YARN application, and thus requires its own slots in the cluster. You must customize the job according tohttps://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/mapred/jobcontrol/Job.html
. -
The Stocator Swift driver (
swift2d://
) is not supported using the MapReduce API. All MapReduce applications that require access to Oracle Cloud Infrastructure Object Storage Classic should make use of the Hadoop OpenStack Swift driver (swift://
).
Use the Cluster Console
You can submit MapReduce jobs from the Jobs tab in the Big Data Cloud Console. See Create a Job.
Use the REST API
You can use the REST API to submit MapReduce jobs.Example job submission:
{ "job": { "applicationClass": "org.apache.hadoop.examples.ExampleDriver", "applicationFile": "hdfs:///mapred/examples/hadoop-mapreduce-examples.jar", "applicationArguments": [ "wordcount", "hdfs:///mapred/data/one_word.txt", "hdfs:///tmp/one_word-1487791745.out" ], "hadoopConf": { "a.hadoop.conf.key": "a.hadoop.conf.value" }, "queue": "api", "applicationName":"MapReduceWordCount" } }
Assuming the above content is contained within payload_mr_job.json
, the corresponding REST API request would look as follows:
curl -k -s -X POST " "https://big_data_cluster_host:1080/bdcsce/api/v1.1/clustermgmt/identity_domain/instances/cluster_name/jobs/mapred >/jobs/mapred" -H "X-ID-TENANT-NAME: identity_domain" -H "Content-Type: application/json; charset=utf-8" --user "bdcsce_admin:csm_password" -d @payload_mr_job.json
For information about using the REST API, see REST API for Oracle Big Data Cloud.
Use the Command Line Interface
MapReduce jobs can be executed from the shell command line. To do so, SSH to any node in the cluster, and then submit the job. The following example shows how to submit a MapReduce job using the command line:
opc@host/>sudo su oracle oracle@host/>hadoop fs -mkdir /user/oracle/mapredsmokeinput oracle@host/>hadoop fs -put /tmp/ambari.properties.1 /user/oracle/mapredsmokeinput oracle@host/>yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-2.*.jar wordcount /user/oracle/mapredsmokeinput /user/oracle/mapredsmokeoutput