Step 1: Configuring the Properties
1. From the DMT Configurations window, set Sqoop Mode as Client.
2. Specify the path of the HDFS working directory for Sqoop related operations in the Sqoop Working Directory field.
3. From the Register Cluster tab in the DMT Configurations window, register a cluster with Target Information domain name as the Cluster Name in case of T2H or register a cluster with Source Information domain name as the Cluster Name in case of H2T.
For details, see DMT Configurations section in the OFS Analytical Applications Infrastructure User Guide.
Step 2: Copy Third Party Jars
Copy the following Third Party Jars from the installation libraries into the $FIC_HOME/ext/lib for either the CDH or Apache Hadoop Server:
CDH 5.++Version
§ avro-1.7.4.jar
§ commons-cli-1.2.jar
§ commons-httpclient-3.1.jar
§ hadoop-hdfs-2.0.0-cdh4.7.0.jar
§ jackson-core-asl-1.8.8.jar
§ jackson-mapper-asl-1.8.8.jar
§ protobuf-java-2.4.0a.jar
§ servlet-api.jar
§ sqoop-test-1.4.3-cdh4.7.0.jar
§ sqoop-1.4.3-cdh4.7.0.jar
§ htrace-core-3.0.4.jar
NOTE |
Version of Jars depends on the CDH Version
and the Drivers used. |
CDH 6.3.x
§ avro-tools-1.8.2-cdh6.3.0.jar
§ commons-cli-1.4.jar
§ commons-httpclient-3.1.jar
§ commons-net-3.1.jar
§ hadoop-hdfs-client-3.0.0-cdh6.3.0.jar
§ hadoop-mapreduce-client-core-3.0.0-cdh6.3.0.jar
§ htrace-core4-4.2.0-incubating.jar
§ jackson-core-2.9.9.jar
§ jackson-mapper-asl-1.9.13.jar
§ protobuf-java-3.1.0.jar
§ re2j-1.1.jar
§ servlet-api-2.5.jar
§ sqoop-1.4.7-cdh6.3.0.jar
§ sqoop-test-1.4.7-cdh6.3.0.jar
NOTE |
· CDH 6.3.x applies only to OFS AAAI Release 8.0.7.2.0.+. · Additional
JARs will be copied during OFSAA Installation with Big Data enabled. · The version of jars to be copied will differ depending upon the version of CDH configured. |
Apache Hadoop
§ hadoop-mapreduce-client-core-3.1.1.jar
§ re2j-1.1.jar
§ commons-cli-1.2.jar
§ hadoop-hdfs-3.1.1.jar
§ hadoop-hdfs-client-3.1.1.jar
§ protobuf-java-2.5.0.jar
§ htrace-core4-4.1.0-incubating.jar
§ commons-net-3.6.jar
§ commons-codec-1.11.jar
§ sqoop-test-1.4.7.jar
§ sqoop-1.4.7.jar
§ jackson-mapper-asl-1.9.13.jar
§ jackson-core-2.3.1.jar
§ avro-1.8.1.jar
NOTE |
Additional JARs are copied during OFSAA Installation with Big Data enabled. See the Configuring Big Data Processing Section in the Oracle Financial Services Advanced Analytical Applications Infrastructure Application Pack Installation and Configuration Guide for a list of the JARs. |
Step 3: Copy Configuration XMLs from Hadoop Cluster
Copy core-site.xml, hdfs-site.xml, mapred-site.xml, hive-site.xml, and yarn-site.xml from the Hadoop Cluster to the Configuration File Path given in the Cluster Configurations window of the registered cluster. Note that only Client Configuration Properties are required.
If Cloudera Manager is used, the same can be downloaded directly which will contain only the client properties.
NOTE |
If proxy user is enabled and the Job is submitted by the same, the user should be created in every node of the Hadoop Cluster. |