HBase

1. Download and extract hbase 1.1.2 stable binary

Hadoop v2.5.2 & HBase v1.1.2 are used.They are compatible with each other.
and JDK used 1.7

2. HBase Setup in standalone mode

[a] Edit conf/hbase-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-oracle/

[b] Edit conf/hbase-site.xml

<configuration>

  <property>

    <name>hbase.rootdir</name>

    <value>file:///home/user/Hadoop/hbase-1.1.2/hbase_data/hbase</value>

  </property>

  <property>

    <name>hbase.zookeeper.property.dataDir</name>

    <value>/home/user/Hadoop/hbase-1.1.2/hbase_data/zookeeper</value>

  </property>

</configuration>

You do not need to create the HBase data directory. HBase will do this for you. If you create the directory, HBase will attempt to do a migration, which is not what you want.

[c] ./bin/start-hbase.sh   &  ./bin/stop-hbase.sh

You can use the jps command to verify that you have one running process called HMaster. In standalone mode HBase runs all daemons within this single JVM, i.e. the HMaster, a single HRegionServer, and the ZooKeeper daemon.

[d] in other shell, can start hbase shell, ./bin/hbase shell

3. HBase Setup Intermediate – Pseudo-Distributed Local Install

Pseudo-distributed mode means that HBase still runs completely on a single host, but each HBase daemon (HMaster, HRegionServer, and Zookeeper) runs as a separate process.

Unless you configure the hbase.rootdir  property in hbase-site.xml , data is still stored in /tmp/ directory OR if any other directory configured in hbase.rootdir.

You can configure Hbase to store data in HDFS as follows :

(assuming you have installed & configured hadoop and HDFS already on your local system and/or a remote system , and that they are running and available. It also assume you are using Hadoop 2.)

1. Stop HBase if it is running.

2. Configure HBase.

Edit the hbase-site.xml configuration. First, add the following property. which directs HBase to run in distributed mode, with one JVM instance per daemon.

<property >

           <name>hbase.cluster.distributed </name>

  <value>true </value>

</property>

Next, change the hbase.rootdir from the local filesystem to the address of your HDFS instance, using the hdfs://// URI syntax. In this example, HDFS is running on the localhost at port 50070.

       

        <property>

          <name>hbase.rootdir</name>

          <value>hdfs://amit.rodge-ubuntu:50070/hbase</value>

        </property>

     

3. Start Hbase.  ./bin/start-hbase.sh

jps command should show the HMaster, HQuorumPeer and HRegionServer processes running.

4. Check Hbase directory in HDFS.

If everything goes fine , Hbase creates its directory in HDFS. As per configuration mentioned above in hbase-site.xml , it create directory named ‘hbase’ in HDFS.

$ hadoop fs –ls  /hbase

Found 7 items

drwxr-xr-x   - hbase users          0 2014-06-25 18:58 /hbase/.tmp

drwxr-xr-x   - hbase users          0 2014-06-25 21:49 /hbase/WALs

drwxr-xr-x   - hbase users          0 2014-06-25 18:48 /hbase/corrupt

drwxr-xr-x   - hbase users          0 2014-06-25 18:58 /hbase/data

-rw-r--r--   3 hbase users         42 2014-06-25 18:41 /hbase/hbase.id

-rw-r--r--   3 hbase users          7 2014-06-25 18:41 /hbase/hbase.version

drwxr-xr-x   - hbase users          0 2014-06-25 21:49 /hbase/oldWALs

 

4. HBase Setup Fully Distributed

            you need a fully-distributed configuration to fully test HBase and to use it in real-world scenarios. In a distributed configuration, the cluster contains multiple nodes, each of which runs one or more HBase daemon. These include primary and backup Master instances, multiple Zookeeper nodes, and multiple RegionServer nodes.

The architecture will be as follows:

Hadoop					Nodes	HBase
NodeName	Secondary NodeName	Resource Manager	Data node	Node Manager		Hmaster	QuorumPeer (Zookeeper)	RegionServer
yes	yes	yes	no	no	MasterNode	yes (primary)	yes	no
no	no	no	yes	yes	SalveNode1	yes (backup)	yes	yes
no	no	no	yes	yes	SalveNode2	no	yes	yes
no	no	no	yes	yes	SalveNode3	no	no	yes
no	no	no	yes	yes	SalveNode4	no	no	yes
no	no	no	yes	yes	SalveNode5	no	no	yes
no	no	no	yes	yes	SalveNode6	no	no	yes
no	no	no	yes	yes	SalveNode7	no	no	yes

We have set up Hbase cluster in parallel to Hadoop cluster.

-          - Region server should not be running on the same machine on which HMaster (primary) is running.

-         - We have used 3 zookeeper machine (quorum peers).

-         - For Hbase cluster to work, only HDFS service should be up and running.

-         -  Hadoop 2.5.2 , Hbase 1.1.2 and JDK 7.

-         -  There is compatibility chart provided for Hbase Vs Hadoop version.

( https://hbase.apache.org/book.html)

Configurations :

  1.  Create file named backup-masters at /home/tmaster/hbase/conf path

  2.     Mention backup master machines hostname

e.g.   slavelarge1

  3.   Mention JAVA_HOME in hbase-env.sh

export JAVA_HOME=/usr/java/latest

  4.  Edit hbase-site.xml.

<configuration>

   <property>

      <name>hbase.rootdir</name>

      <value>hdfs://tmaster:9000/hbase</value>

   </property>

   <property>

      <name>hbase.zookeeper.property.dataDir</name>

      <value>/home/tmaster/hbase/hbase_data/zookeeper</value>

   </property>

   <property>

      <name>hbase.cluster.distributed</name>

      <value>true</value>

   </property>

   <property>

      <name>hbase.zookeeper.quorum</name>

      <value>tmaster,slavelarge1,slavelarge2</value>

   </property>

</configuration>

  5.   /home/tmaster/hbase/conf/regionservers file.

slavelarge1

slavelarge2

slavelarge3

slavelarge4

slavelarge5

slavelarge6

slavelarge7

  6.   All machines should maintain same configurations across all nodes.

  7.   Environment variables set

export HBASE_HOME=/home/tmaster/hbase

export CLASSPATH=$CLASSPATH:/home/tmaster/hbase/conf

  8.  When Hbase starts using command : start-hbase.sh , stopping stop-hbase.sh

It creates folder  named hbase in hdfs . it ensures hbase configurations is correct and up & running.

  9.  Processes running on nodes.

a.       Master node (jps)

2945 SecondaryNameNode

3095 ResourceManager

3593 HQuorumPeer

3690 HMaster

2749 NameNode

3855 Jps

b.      Slavelarge1 (jps)

2964 HMaster

3593 HQuorumPeer

2856 HRegionServer

2694 NodeManager

2584 DataNode

3180 Jps

c.       All other nodes slavelarge2-slavelarge7(jps)

3077 Jps

2694 NodeManager

2856 HRegionServer

2584 DataNode

Comparison of Hbase Vs Hive:

http://blog.cloudera.com/blog/2011/02/log-event-processing-with-hbase/

Hadoop = HDFS + Computation framework (MapReduce).

HDFS lacks random read and write access.

HBase has nothing to do with Hadoop computation framework.

- Is database (NoSQL).stores data as key/value pairs.Its distributed , scalable , big data store

 -Depends on HDFS not on MapReduce.

- Is online processing system.

- Real time querying (get/set/update request processing)

- Good for lookup use cases (random read/write) but not good for queries need to access all data.

- Used  as a back-end for high throughput systems.

 - Can write map-reduce jobs against HBase.

 

Hive - provide data warehousing. Facilitates on top of existing Hadoop cluster.

- Is never a database.

- SQL like interface (SQL like layer on HDFS and map-reduce)

- Depends on MapReduce(batch processing) + HDFS.

- MapReduce based Analysis/ Summarization tool running on Top of Hadoop.

- Doesn't facilitates real time queries anywhere close to real time and row level updates.

- Doesn't support updating data set.(large analytical processing application usage).

- Best used for batch jobs over large sets of immutable data (like web logs).

- Can map existing HBase tables to Hive and operate on them.

- HDFS itself not good at random retrievals of single file records, something you’ll notice if you try and return a single row from a Hive table request.

 Moreover, HDFS files are write-once, no updates or overwrites, which is why Hive only supports SELECTS and not UPDATES or DELETES.

  Log event processing scenarios share a common set of requirements:;

  1. Data must be ingested into the system incrementally – one day or so worth of data at a time.

  2. Data is processed at a variety of time scales. Daily reporting often cares only about one day’s worth of data, while machine learning applications may require digesting several months worth of data to build models.

  3. Some events are naturally associated with others. An ad click is logged separately from an ad impression, but the two need to be processed together.  Some data extraction applications need to process these associated events together, but others only care about the individual events.

  4. Random-access lookups to track all the actions of a user across time are often helpful.

  This is a powerful debugging tool to understand how the user interacts with the web at large and with the whole system in particular.

 HDFS (i.e. using HIVE) is great for landing and then processing large chunks of data, if you’re looking for more granular, database-type storage on Hadoop, you’ll need to think of something else other than HIVE. In context of Hadoop, other thing is HBASE , a "NoSQL" database. 

 Reason for choosing HBASE:

 If we then want to do fast individual record lookups within the landed data. HBase’s support for complex record types, making it easy to store for example nested XML datasets,and its ability to hold completely different sets of “columns” for each row in the database and  even version those rows giving us almost a “multi-dimensional” database. Internally, HBase stores data as key-value pairs giving it the ability to hold completely different data in each database row and  under the covers HBase data is in turn stored in indexed “StoreFiles” within HDFS,giving it HDFS’s scalability and access to the Hadoop framework, but adding fast random access to individual records.

Column families are a compromise between row-oriented vs. column-oriented access. To extend hosted web page example, a row access would fetch all data (columns) for a single web site. An example of a column-oriented operation would be to sum the number of page views across all sites. The latter operation does not require the browser and connection details, which are much larger than the numeric values for view counts and would significantly affect query performance.

Therefore, HBase provides column families as an optimization that supports column operations.

As to whether or not the columns should be in the same table... I would just follow normal data modelling guidelines and put all the columns in the same table if they are attributes of the same entity. Column families are about performance not schema.

  

Even if we set number of Reducers to zero, HFileOutputFormat initiates a reducer task to sort and merge the mapper output to make this file HTable compatible. The number of reducers is equal to number of regions in HBase table.

Find a sample code to prepare data for HBase bulk load via a MapReduce job,

here : http://databuzzprd.blogspot.in/2013/11/bulk-load-data-in-hbase-table.html

Links :
http://www.slideshare.net/hmisty/20090713-hbase-schema-design-case-studies

<! -- 
Exception in thread "main" org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region
1.
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-0.98.16.1-hadoop2.jar importtsv '-Dimporttsv.seperator=,' -Dimporttsv.columns=HBASE_ROW_KEY,m:v1,m:v2 -Dimporttsv.bulk.output=hdfs://tmaster:9000/test/output importTest hdfs://tmaster:9000/test/input/file1.txt
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-0.98.16.1-hadoop2.jar importtsv '-Dimporttsv.seperator=,' -Dimporttsv.columns=HBASE_ROW_KEY,m:v1,m:v2 -Dimporttsv.bulk.output=hdfs://tmaster:9000/hbase/example/output importTest hdfs://tmaster:9000/hbase/example/input/file1.txt
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-1.1.2.jar importtsv '-Dimporttsv.seperator=,' -Dimporttsv.columns=HBASE_ROW_KEY,m:v1,m:v2 -Dimporttsv.bulk.output=hdfs://tmaster:9000/test/output importTest hdfs://tmaster:9000/test/input/file1.txt

HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-1.1.2.jar completebulkload -Dimporttsv.bulk.output=hdfs://tmaster:9000/hbase/data/default/importTest/8f3e61d9eab52e3965c33040b21fb61f importTest

2.
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-1.1.2.jar importtsv '-Dimporttsv.seperator=,' -Dimporttsv.columns=HBASE_ROW_KEY,c:,m:v2 -Dimporttsv.bulk.output=hdfs://tmaster:9000/cflogs/output importTest hdfs://tmaster:9000/cflogs/input/
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-1.1.2.jar completebulkload -Dimporttsv.bulk.output=hdfs://tmaster:9000/hbase/data/default/importTest/8f3e61d9eab52e3965c33040b21fb61f importTest


HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-0.98.16.1-hadoop2.jar completebulkload -Dimporttsv.bulk.output=hdfs://tmaster:9000/test/output importTest
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-0.98.16.1-hadoop2.jar completebulkload -Dimporttsv.bulk.output=hdfs://tmaster:9000/hbase/example/output importTest

hadoop jar /home/tmaster/hbase/lib/hbase-server-0.98.16.1-hadoop2.jar completebulkload [-c /home/tmaster/hbase/conf/hbase-site.xml] hdfs://tmaster:9000/test/output importTest

create '', {NAME=>'m'}

HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar hbase-stuff/hbase-cflogs-import.jar com.samsung.hadoop.hbase.bulkimport.Driver hdfs://tmaster:9000/test/cflog-input/ hdfs://tmaster:9000/test/cflog-output qa_cflog_table
 
-->