Next, obtain the Hadoop client libraries and put them on the Admin
Server.
BDD requires a number of client libraries to interact with Hadoop. In
a normal Hadoop cluster, these libraries are spread out, making it difficult
for BDD to find them all. To solve this issue, the upgrade script adds the
required libraries to a single JAR, called the Hadoop fat JAR, and distributes
it to all BDD nodes.
The specific libraries you need depend on your Hadoop distribution.
The location you put them in is arbitrary, as you will define it in
bdd.conf.
- CDH: Download the
following files from
http://archive-primary.cloudera.com/cdh5/cdh/5/ to the
Admin Server and extract them:
- spark-<spark_version>.cdh.<cdh_version>.tar.gz
- hive-<hive_version>.cdh.<cdh_version>.tar.gz
- hadoop-<hadoop_version>.cdh.<cdh_version>.tar.gz
- avro-<avro_version>.cdh.<cdh_version>.tar.gz
Be sure to download the files that correspond to the component
versions you currently have installed.
- HDP: Copy the
following libraries from your Hadoop nodes to the Admin Server. Note that these
directories might not all be on the same node.
- /usr/hdp/<version>/hive/lib/
- /usr/hdp/<version>/spark/lib/
- /usr/hdp/<version>/hadoop/
- /usr/hdp/<version>/hadoop/lib/
- /usr/hdp/<version>/hadoop-hdfs/
- /usr/hdp/<version>/hadoop-hdfs/lib/
- /usr/hdp/<version>/hadoop-yarn/
- /usr/hdp/<version>/hadoop-yarn/lib/
- /usr/hdp/<version>/hadoop-mapreduce/
- /usr/hdp/<version>/hadoop-mapreduce/lib/
- MapR: Locate the
following directories on your Hadoop nodes and copy them to the Admin Server.
Note that they might not all be on the same node.
-
/opt/mapr/spark/spark-1.6.1/lib
- /opt/mapr/hive/hive-1.2/lib
- /opt/mapr/zookeeper/zookeeper-3.4.5
- /opt/mapr/zookeeper/zookeeper-3.4.5/lib
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/tools/lib
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn
- /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib
If you're upgrading from 1.1.0 or 1.1.1, you should apply the upgrade
hotfix now. If you have a later version, move on to
Backing up your current cluster.
