This is a simple example of how Hadoop on Ubuntu Linux can be upgraded from V1 to Yarn. It shows the steps, the configuration, a mapreduce check and the errors encountered.

1. Apache Yarn Upgrade
● Example upgrade
● From V1 -> Yarn
● Environment
● Approach
● Install steps
● Install check
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

2. Yarn Upgrade Environment
● Java OpenJDK 1.6.0_27
● Ubuntu 12.04
● Maven 3.0.4
● Hadoop 1.2.0
● Mahout 0.9
● Hadoop to install
– 2.0.6-alpha
Full details are available from our web site site
under guides folder
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

3. Yarn Upgrade Approach
● Install along side existing Hadoop on all nodes
● Use existing hdfs
● Change cfg files on all nodes
● Set up as single nodes and test via mapreduce
● Create cluster and test via mapreduce
● Check web GUI access
Full details are available from our web site site
under guides folder
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

4. Yarn Upgrade Install
● Build with Maven into a distribution directory
mvn clean package -Pdist -Dtar -DskipTests -Pnative
release created under ./hadoop-dist/target/hadoop-2.0.6-alpha
● Only skip tests after first build to speed things up
● Configure $HOME/.bashrc
– HADOOP_COMMON_HOME
– HADOOP_HDFS_HOME
– HADOOP_MAPRED_HOME
– HADOOP_YARN_HOME
– HADOOP_CONF_DIR
– YARN_CONF_DIR
– MAPRED_CONF_DIR
– HADOOP_PREFIX
– PATH
– YARN_CLASSPATH
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

5. Yarn Upgrade Install
● Set up core-site.xml
cd $HADOOP_COMMON_HOME/etc/hadoop
● Alter values for
– fs.default.name
– hadoop.tmp.dir
– fs.checkpoint.dir
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

6. Yarn Upgrade Install
● Set up hdfs-site.xml
cd $HADOOP_HDFS_HOME/etc/hadoop
● Alter values for
– dfs.name.dir
– dfs.data.dir
– dfs.http.address
– dfs.secondary.http.address
– dfs.https.address
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

7. Yarn Upgrade Install
● Set up yarn-site.xml
cd $YARN_CONF_DIR
● Alter values for
– yarn.resourcemanager.resource-tracker.address
– yarn.resourcemanager.scheduler.address
– yarn.resourcemanager.scheduler.class
– yarn.resourcemanager.address
– yarn.nodemanager.local-dirs
– yarn.nodemanager.address
– yarn.nodemanager.resource.memory-mb
– yarn.nodemanager.remote-app-log-dir
– yarn.nodemanager.log-dirs
– yarn.nodemanager.aux-services
– yarn.web-proxy.address
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

8. Yarn Upgrade Install
● Set up mapred-site.xml
cd $MAPRED_CONF_DIR
● Alter values for
– mapreduce.cluster.temp.dir
– mapreduce.cluster.local.dir
– mapreduce.jobhistory.address
– mapreduce.jobhistory.webapp.address
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

9. Yarn Upgrade Install
● Set up capcity-scheduler.xml
cd $HADOOP_YARN_HOME/etc/hadoop
● Alter values for
– yarn.scheduler.capacity.maximum-applications
– yarn.scheduler.capacity.maximum-am-resource-percent
– yarn.scheduler.capacity.resource-calculator
– yarn.scheduler.capacity.root.queues
– yarn.scheduler.capacity.child.queues
– yarn.scheduler.capacity.child.unfunded.capacity
– yarn.scheduler.capacity.child.default.capacity
– yarn.scheduler.capacity.root.capacity
– yarn.scheduler.capacity.root.unfunded.capacity
– yarn.scheduler.capacity.root.default.capacity
– yarn.scheduler.capacity.root.default.user-limit-factor
– yarn.scheduler.capacity.root.default.maximum-capacity
– yarn.scheduler.capacity.root.default.state
– yarn.scheduler.capacity.root.default.acl_submit_applications
– yarn.scheduler.capacity.root.default.acl_administer_queue
– yarn.scheduler.capacity.node-locality-delay
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

10. Yarn Upgrade Install
● Start Resource Manager
cd $HADOOP_YARN_HOME
sbin/yarn-deamon.sh start resourcemanager
● Start Node Manager
cd $HADOOP_YARN_HOME
sbin/yarn-deamon.sh start ndemanager
● Test via map reduce job
cd $HADOOP_MAPRED_HOME/share/hadoop/mapreduce
$HADOOP_COMMON_HOME/bin/hadoop jar
hadoop-mapreduce-examples-2.0.6-alpha.jar randomwriter out
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

11. Yarn Upgrade Install
● Mapreduce job should end with
BYTES_WRITTEN=1073750341
RECORDS_WRITTEN=102099
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=1085699265
Job ended: Sun Aug 25 12:45:35 NZST 2013
The job took 89 seconds.
● Run this test on each node being upgraded
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

12. Yarn Upgrade Install
● Stop the servers
cd $HADOOP_YARN_HOME
sbin/yarn-daemon.sh stop resourcemanager
stopping resourcemanager
sbin/yarn-daemon.sh stop nodemanager
stopping nodemanager
● Alter Hadoop env
cd $HADOOP_CONF_DIR
vi hadoop-env.sh
add a JAVA_HOME definition at the end. i.e.
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

13. Yarn Upgrade Install
● Alter $HADOOP_CONF_DIR/slaves file
– Add details ( one per line ) for slave nodes
● Format the cluster
– DONT have the cluster running else you will lose data
– hdfs namenode -format
● Now proceed to start the cluster
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

14. Yarn Upgrade Install
cd $HADOOP_COMMON_HOME
sbin/hadoop-daemon.sh --config $HADOOP_COMMON_HOME/etc/hadoop
--script hdfs start namenode
cd $HADOOP_COMMON_HOME
sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
cd $HADOOP_YARN_HOME
sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
cd $HADOOP_YARN_HOME
sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
cd $HADOOP_YARN_HOME
bin/yarn start proxyserver --config $HADOOP_CONF_DIR
cd $HADOOP_MAPRED_HOME
sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

15. Yarn Upgrade Install
● Use jps to check servers running
jps
5856 DataNode
6434 Jps
5776 NameNode
6181 NodeManager
6255 WebAppProxyServer
5927 ResourceManager
6352 JobHistoryServer
● Then run the same mapreduce job on the cluster
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

16. Web Access
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

17. Web Access
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

18. Web Access
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

19. Contact Us
● Feel free to contact us at
– www.semtech-solutions.co.nz
– info@semtech-solutions.co.nz
● We offer IT project consultancy
● We are happy to hear about your problems
● You can just pay for those hours that you need
● To solve your problems