As a man among men: October 2015

This section contains instructions for Hadoop installation on ubuntu.
This is Hadoop quickstart tutorial to setup Hadoop quickly. steps to install Hadoop in Pseudo distributed mode (single node cluster)
In this tutorial, I will describe required steps for deploying Hadoop.
The main goal of this tutorial is to get a ”simple” Hadoop installation up and running on AWS with input as s3 bucket and hdfs.so that you can play around with the software and learn more about it.
We mainly learn how to setup hadoop a single node cluster in Pseudo-Distributed Operation.

Ubuntu 14.0 (LTS) AMI from EC2
Hadoop 2.7.1
JDK 8

[0] AWS EC2 Instance setup

-Create EC2 new instance from available AMI of ubuntu 14.0 (LTS).
-Create .pem credential file for login to EC2 instance.and download it.
-Create hmaster user account (without password) on it.
$ useradd hmaster
(By default root account named 'ubuntu' OR 'ec2-user' used to login to ec2 instance)
this is to control user privileges without password.so only admin i.e. ubuntu can control on it.
-By Default, AMI is configured with public key for 'ubuntu' root user, you can able to login to EC2 instance for 'ubuntu' user with .pem file.
e.g. ssh -i xxx.pem ubuntu@xxx.xxx.xxx.xxx
But same cannt be done user account 'hmaster' as public key of the same .pem file is not configured for hmaster account.
To do this , please follow the link
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#retrieving-the-public-key
It needs to create .ssh folder and .ssh/authorized_keys file to save public key retrieved from .pem file.
.pem would have private key with which you can login for hmaster account also.
e.g. ssh -i xxx.pem hmaster@xxx.xxx.xxx.xxx
- add hmaster as sudoers (edit etc/sudoers) to install any software on it.
- set user hmaster password.
$ passwd hmaster

$ install ssh
$ sudo apt-get install ssh
$ sudo apt-get install rsync

ssh localhost , if it doesn't work
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[1] Install java

$ sudo add-apt-repository ppa:openjdk-r/ppa
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jdk

$ sudo update-alternatives --config java

[2] Environment Variables

update environment variables in .bashrc
export HADOOP_PATH=/home/hmaster/hadoop-2.7.1
export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::")
export PATH=$PATH:$HADOOP_PATH/sbin:/home/hmaster/hadoop-2.7.1/bin

[3] Hadoop Setup

Download Hadoop 2.7.1 version from Apache Hadoop site.
If you want to have java version specific to hadoop installation , JAVA_HOME can be removed from .bashrc file
and set JAVA_HOME in etc/hadoop/hadoop-env.sh
# set to the root of your Java installation
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
OR
you can write environment shell script which includes required environment variables defined in .bashrc file.
Run this file first on linux terminal before running hadoop sothat required environment is create only for that running terminal.
And it dont disturb other applications which want to use different JAVA version.

[6] update hostname

update this hostname same as core-site.xml.
$ sudo vim /etc/hosts
172.31.2.222 hadoop-master
(172.31.2.222 private ip on EC2)

$ hostname hadoop-master
This will update hostname field wherever required in system config file.

[7] Configurations :

[a] Using system memory as HDFS:

$ vi conf/core-site.xml
This file contains the configuration settings for Hadoop Core such as I/O settings that are common to HDFS and MapReduce.

[b] Using S3 as input for mapreduce jobs and local memory as HDFS:

$ vi conf/core-site.xml

This file contains the configuration settings for Hadoop Core such as I/O settings that are common to HDFS and MapReduce.
(https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html)

[c] common for approach [a] and [b] :

1) Edit etc/hadoop/slaves , to add new hostname created i.e. 'hadoop-master' by removing 'localhost'
Rest of the configuration file are same.

2) In /etc/hadoop_env.sh
     export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
     export HADOOP_CLASSPATH=$HADOOP_HOME/share/hadoop/tools/lib/*

3)
$ vi conf/hdfs-site.xml

$ vi conf/mapred-site.xml

( if you want to run a MapReduce job on YARN in a Single Node , recommended for hadoop latest version)

[
    $ hadoop classpath
   /home/hmaster/hadoop-2.7.1/etc/hadoop:/home/hmaster/hadoop-2.7.1/share/hadoop/common/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/common/*:/home/hmaster/hadoop-2.7.1/share/hadoop/hdfs:/home/hmaster/hadoop-2.7.1/share/hadoop/hdfs/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/hdfs/*:
   /home/hmaster/hadoop-2.7.1/share/hadoop/yarn/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/yarn/*:/home/hmaster/hadoop-2.7.1/share/hadoop/mapreduce/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/mapreduce/*:/home/hmaster/hadoop-2.7.1/share/hadoop/tools/lib/*:/home/hmaster/hadoop-2.7.1/ contrib/capacity-scheduler/*.jar]

$ vi etc/hadoop/yarn-site.xml
(if you want to run a MapReduce job on YARN in a Single Node , recommended for hadoop latest version)

[8] Execution

As hadoop/sbin path has been set in environment (.bashrc file).
hadoop can be started from anywhere now.
$ start-all.sh
$ jps
578 NameNode
2100 Jps
1140 ResourceManager
969 SecondaryNameNode
1467 NodeManager
751 DataNode

Now everything is running correctly.

Browse the web interface for the NameNode; by default it is available at:
NameNode - http://localhost:50070/
ResourceManager - http://localhost:8088/

The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs). The newly created folder at ~/hadoop-2.7.1/hadoop_data is created to keep tmp folder. tmp folder creates inside hadoop_data here by hadoop (we dont need to create, just mention path) and the same path has been configured in core-site.xml at /home/hmaster/hadoop-2.7.1/etc/hadoop

FAQ for Errors:

1) you could see the errors like JAVA_HOME is not set when hadoop started. so hadoop doesn't start properly.
OOPs ! , why this error again as I have already set JAVA_HOME in .bashrc file!
Hadoop does ssh to localhost while starting all relevant process.after doing ssh , its not able to get JAVE_HOME variable value to hadoop process, hence this error .so you got 2 options to solve the problem
   a) you can set JAVA_HOME path in enviornment variable such that it can be accessed
     across user logins.
     you can add the path in /etc/environment system file
$ sudo /etc/environment

b) localise the JAVA_HOME path in one of the hadoop script files.
you can edit hadoop/etc/hadoop-env.sh and assign JAVA_HOME path
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Once this path is set, you can start hadoop.

2) To ensure hadoop started correctly. you can run following command
hmaster@ip-172-31-2-222:~$ jps
23096 ResourceManager
23737 Jps
22746 DataNode
22939 SecondaryNameNode
23423 NodeManager

It shows ResourceManager,DataNode,SecondaryNameNode,NodeManager but there is no NameNode process. You can run following command to resolve this
$ hadoop namenode -format

3) If you are using AWS s3 as a input and output both ,then you need to ensure few things
    a) Refer core-site.xml explained in section 7.b , it has mentioned s3,s3n,s3a file-system and its
    related credential AWS accesskeyid and secretkey.
    (note: Amazon claims all 3 types of FS works, but It failed for me for s3 even if I followed process.
    But rest of 2 worked. so I am explaining for other 2 types).

    b) you can see if the s3 related filesystem .jar /lib getting invoked or working correctly when executed s3 path
    e.g.
    $ hadoop fs -ls s3n://hadoop-fs/input/   OR hdfs dfs -ls s3a://hadoop-fs/input/
    $ hadoop fs -ls s3a://hadoop-fs/input/

    If It does not work, and throws exception (class not found), It means the CLASSPATH is not set properly.

    c) hadoop classpath can be found using following command
    $ hadoop classpath
    /home/hmaster/hadoop-2.7.1/etc/hadoop:/home/hmaster/hadoop-2.7.1/share/hadoop/common/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/common/*:/home/hmaster/hadoop-2.7.1/share/hadoop/hdfs:/home/hmaster/hadoop-2.7.1/share/hadoop/hdfs/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/hdfs/*:/home/hmaster/hadoop-2.7.1/share/hadoop/yarn/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/yarn/*:/home/hmaster/hadoop-2.7.1/share/hadoop/mapreduce/lib/*:/home/hmaster/hadoop-2.7.1/share/hadoop/mapreduce/*:/home/hmaster/hadoop-2.7.1/contrib/capacity-scheduler/*.jar

    d) Try to get the *aws*.jar paths
      [ec2-user@ip-172-31-24-18 hadoop-2.7.1]$ find . -name *.jar | grep -i aws
        ./share/hadoop/tools/lib/hadoop-aws-2.7.1.jar
        ./share/hadoop/tools/lib/aws-java-sdk-1.7.4.jar

    e) We added this path /share/hadoop/tools/lib/* in etc/hadoop-env.sh.
    [if you set in hadoop-env.sh , its easy for distribution. else you can update in .bashrc or environment file]
    export HADOOP_CLASSPATH=$HADOOP_HOME/share/hadoop/tools/lib/*

    f) Restart hadoop for any change in configuration.

[9] Hadoop Stop

Hadoop all components can be stopped by command:
$ stop-all.sh

[10] Important: you cannot use S3 as a replacement for HDFS

You cannot use any of the S3 filesystem clients as a drop-in replacement for HDFS. Amazon S3 is an "object store" with
   a) eventual consistency: changes made by one application (creation, updates and deletions) will not be visible until some undefined time.
   b) s3n and s3a: non-atomic rename and delete operations. Renaming or deleting large directories takes time proportional to the number of entries
   -and visible to other processes during this time, and indeed, until the eventual consistency has been resolved.
S3 is not a filesystem. The Hadoop S3 filesystem bindings make it pretend to be a filesystem, but it is not. It can act as a source of data,
and as a destination -though in the latter case, you must remember that the output may not be immediately visible.

So if you see our configuration above, we have mentioned AWS key and value in core-site.xml and name node is started on EC2 instance only.
we didnt configure S3 bucket as namenode. dont get confuse. S3 bucket can be used as input for executing MR job and output of this job is stored back again in S3.

[11] Why to use s3 instead of HDFS? whats the benefit fot it?

(courtesy : http://www.technology-mania.com/2012/05/s3-instead-of-hdfs-with-hadoop_05.html )
Let’s try to understand what the exact problems are & why HDFS is not used in cloud. When new instances are launched on the cloud to build a Hadoop cluster they do not have any data associated with them. So one approach is to copy the entire huge dataset on them, which is not feasible due to various reasons including bandwidth, time to copy & associated cost. Secondly after completion of jobs once again you will
need to copy the result back before terminating cluster machines otherwise the result will be lost when instances are terminated & you will not get anything. Also due to associated cost running the entire cluster just for data collection is not feasible.
As S3 is a storage sevice it offers a benefit to store & accumulate data on Amazon cloud without running any machine which can be then processed through MapReduce later. MapReduce can work with any Hadoop compatible file system & S3 fulfills that compatibility criteria. Apache Hadoop itself provides support for using S3. For MapReduce processing input is taken from S3 & output is again stored on to S3 which can be taken from there anytime later even after terminating cloud instances that processed the data.
BUT the point to be noted here is that this capability comes at the cost of data non localization. In HDFS data is stored local to machines while in S3 data first needs to be moved to individual system of cluster before they can process it & after processing again needs to be stored at S3 which is not as local as local file stem on machine. The data is actually transferred from S3 to local nodes it beats the Haddop philosophy moving computation to data is cheaper than moving data to computation. Although a very high speed network in same environment bridges this gap to a lot of extent.
S3 speed might not be that much good external to Amazon cloud but internally with Amazon cloud it must be good. And this is also one reason why it is challenge to run MapReduce on other clouds as in same cloud environment they don’t have any Hadoop compatible file system like S3 is there in Amazon cloud environment.

So this is using S3 as a data store only. Which means that the job will first load data from S3 and bring it in local file system of the cluster
(A huge IO effort on huge amount of data) and then run the map jobs and reduce jobs and then write the results back into S3 (another high amount of IO over a relatively slow network).

[12] References:

https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
http://www.datasciencecentral.com/profiles/blogs/s3-as-input-or-output-for-hadoop-mr-jobs
http://stackoverflow.com/questions/4092852/i-cant-get-hadoop-to-start-using-amazon-ec2-s3
https://wiki.apache.org/hadoop/AmazonS3
https://letsdobigdata.wordpress.com/2014/01/13/setting-up-hadoop-1-2-1-multi-node-cluster-on-amazon-ec2-part-2/
http://www.eduonix.com/blog/bigdata-and-hadoop/a-step-by-step-guide-to-install-hadoop-cluster-on-amazon-ec2/

[13] Execute Hadoop MR job

a) Running MR job for input from HDFS directory
$ hadoop jar hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep /input output/ 'dfs[a-z.]+'
$ hadoop jar twitter_example.jar /input/ output/

b) Running MR job for input from AWS s3 bucket
$ hadoop jar hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep s3n://hadoop-fs/input-word/ s3n://hadoop-fs/output/ 'dfs[a-z.]+'
$ hadoop jar hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep s3a://hadoop-fs/input-word/ s3a://hadoop-fs/output/ 'dfs[a-z.]+'
$ hadoop jar twitter_example.jar s3a://hadoop-fs/input/ s3a://hadoop-fs/output/

FAQ:
    [1] If it throws error or exception for MR job (accepting input from S3 bucket) as
    Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
    There goes classpath missing from hadoop yarn configuration. Earlier we had configured classpath for FS shell command execution.
    you need to update hadoop classpath in mapred-site.xml mentioned in 7.c.3

c) Input can be taken from S3 bucket and output can be HDFS directory for performance issue of accessing S3 bucket.
Once MR job is completed output can be copied to AWS s3 bucket.

$ hadoop distcp s3://accessKeyid:secretkey@bucket/ hdfs://CDH5-nameservice/ .

[14] An alternative to this Amazon provides EMR cluster to run hadoop applications.

http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-common-programming-sample.html

node.js allows you to build scalable network applications using JavaScript on the server side
Its single threaded server

Setup & basic example:

Download node.js code
Compile with prefix path
make and make install
set environment path for node in profile information
go to node install dir, it has 'node' and 'npm' binaries
npm is for packaging or resolving package dependencies

Basic example:
any server can be run as
$> node app.js
[ e.g. app.js ->
var http = require('http');
http.createServer(function(req,res){
res.writeHead(200,{'Content-Type':'text/plain'});
res.end('Hello node.js\n');
}).listen(8124,"127.0.0.1");
console.log('Server running at http://127.0.0.1:8124/');
]

if you want to create new app package , one example at below link
https://stormpath.com/blog/build-nodejs-express-stormpath-app/

Advantages:

I/O latency is crucial for implementing server.its not cpu bound operation. Most of the time , it goes in waiting for response.
so generally implementation is scaled with threading model( by running more threads concurrently for each request)
But for this, we will start getting following problems
-Context Switching OverHead
-Execution stacks take up memory
-Complicates concurrency

Some platforms used processes instead of threads to solve above problems
But it again it gets into
-High memory usage
-Process scheduling overhead

Node Internals:

Node Implementation scales with Event Loop
-Handles many concurrent requests in one process/thread
With this, it will get all i/o operations out of the way and gets into cpu bound operations.

                    | |                __________
                           |         |                | filesystem|
Event Queue              |              |             |----------------|
__________        |               |        | network |
|__________|------>    | Event Loop    |--------> |---------------|
|__________|          |              |            | process |
|__________|        ^|            |           |---------------|
|__________|              |        |                 | other      |
|__________|                 | |        |_________|
     ^|                 Thread of node                              |    Thread Pool of underlying O.S.
    |_______________________________________|

Node Platform
                    node standard library (c code)
        ------------------------------------------------
                    node bindings (c code)
                    (http,socket, file system)
        ------------------------------------------------
        V8 | thread pool| event loop | crupto | DNS
    (c code)| (libeio)   | (libev)   | OpenSSL)| (c-ares)

Node is asynchronous even driven architecture. It has even loop through which it process the
events. Its single thread model.

Why JavaScript ?
- JavaScript has certain characteristics that make it very different than other dynamic languages, namely that it has
no concept of threads.Its model of concurrency is completely based around events.

While executing the code written in js, Node registers the events. and then Node goes in event loop(i.e. checking for events continuously).
When the particular event triggers (i.e. request event,response,connection,close etc.), Node runs the corresponding callback to take further action in application.

How Inheritance can be achieved in NodeJS example:

file1 :-> apirequest.js

var EventEmitter = require('events').EventEmitter;
var util = require('util');
function APIRequest(endpoint) { }
util.inherits(APIRequest, EventEmitter);
APIRequest.prototype.start = function() { }
module.exports = APIRequest;

file2 :-> feed.js

var util = require('util');
var APIRequest = require('../lib/api_request');

function Feed(endpoint) {
APIRequest.call(this, endpoint);
}
util.inherits(Feed, APIRequest);
var feed = new Feed(endpoint);
feed.start();

how to resolve dependent packages ?

1) npm -> node package manager , dependent packages of existing application/package can be downloaded
using npm , it stores in global node_modules in node\lib\node_modules\npm\node_modules.
you need to use command require() to use in your application.
most of the packages are downloaded from github.
command -> npm install ( this access https://github.com to download pkgs)

2) If you want to manually install the dependency pkg. e.g aws-sdk.
visit github site and download required pkg. you can keep the code in global node_modules i.e
node\lib\node_modules\npm\node_modules. and update package.json kept at node\lib\node_modules\npm
accordingly to be used by application. and resolve the dependent pkgs accordingly.

3) If you don't want to keep it in global node_modules , you can create local node_modules folder inside
your application folder and keep the downloaded packages inside it. npm search algorithm locates 'node_modules'
folder in its parent directory hierarchy in turn which has relevant packages.
you can browser npm search alogo @ http://www.bennadel.com/blog/2169-where-does-node-js-and-require-look-for-modules.htm

4)To resolve packages by name and version, npm talks to a registry website that implements the CommonJS Package Registry specification for reading package info.
The official public npm registry is at http://registry.npmjs.org/.

npm config set registry http://registry.npmjs.org/
npm install
OR
npm install --registry http://registry.npmjs.org/
If there is issue with SSL i.e. behind any corporate proxy/proxy which is causing problem for downloading packages.
you can set the registry to resolve packages.
This would download the required package and keep it 'node_modules' folder locally inside an application or in existing node_modules folder in parent hierarchy.

References:

(https://www.youtube.com/watch?v=L0pjVcIsU6A)
Node uses JavaScript as language to create servers. JavaScript v8 is better dynamic language compared to others.
Of Course, this is not faster than native language as c,java but faster than other dynamic language python,php,ruby etc..
http://benchmarksgame.alioth.debian.org/

As a man among men

Friday, October 30, 2015

Hadoop cluster setup on AWS EC2 instance => input & output with S3 for MapReduce