Apache Hadoop installation with Cloudera

Working on doc:
18.03.2020


You can download prapared 1 node cloudera cluster from cloudera website, and just run it in your virtual machine.


Can be interesting:

Configuring Big Data Cluster using Cloudera Manager
https://www.youtube.com/watch?v=jKyJmPPmm1A


On Host Machine

$ sudo vi /etc/hosts
#---------------------------------------------------------------------
# Cloudera cluster
#---------------------------------------------------------------------

192.168.0.10 master.cloudera master
192.168.0.11 node1.cloudera node1
192.168.0.12 node2.cloudera node2
192.168.0.13 node3.cloudera node3


$ vagrant plugin install vagrant-hostmanager


master - 8 GB RAM
3 nodes - 2 GB RAM


With 4 GB RAM for master node there were errors on installation.


$ mkdir ~/vagrant-hadoop-cloudera && cd ~/vagrant-hadoop-cloudera

$ git clone https://github.com/matematika-org/vagrant-hadoop-cloudera .

$ cd centos7

$ vagrant box update

$ vagrant up

$ vagrant status
Current machine states:

master.cloudera             running (virtualbox)
node1.cloudera              running (virtualbox)
node2.cloudera              running (virtualbox)
node3.cloudera              running (virtualbox)


$ vagrant ssh master.cloudera

$ sudo su -


All nodes

# {
    echo never > /sys/kernel/mm/transparent_hugepage/defrag
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
}

and

# vi /etc/hosts
remove

127.0.0.1       master.cloudera master


Install Cloudera Manager (master.clouder)

# cd /tmp

// cm6
# wget https://archive.cloudera.com/cm6/6.3.1/cloudera-manager-installer.bin

also possible

// cm5
# wget https://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

or

// cm7
# wget https://archive.cloudera.com/cm7/7.0.3/cloudera-manager-installer.bin


# chmod +x cloudera-manager-installer.bin

# ./cloudera-manager-installer.bin

Next --> Next --> Yes ... etc ...


$ service cloudera-scm-server status


// wait 2-3 minutes

$ curl -i -u 'admin:admin' http://localhost:7180/api/v1/tools/echo


even better:

$ while true; do curl -i -u 'admin:admin' http://localhost:7180/api/v1/tools/echo; sleep 10; done


response should be:

HTTP/1.1 200 OK
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: CLOUDERA_MANAGER_SESSIONID=b3ywt3my169unctxtycduab7;Path=/;HttpOnly
Content-Type: application/json
Date: Mon, 16 Mar 2020 23:49:21 GMT
Transfer-Encoding: chunked
Server: Jetty(6.1.26.cloudera.4)

{
  "message" : "Hello, World!"
}


http://master.cloudera:7180/

admin/admin


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Add nodes:


master.cloudera
node1.cloudera
node2.cloudera
node3.cloudera


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


LOGIN/PASSWORD
root/root


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Resume

Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Fallback Erasure Coding Policy - No Default Erasure Coding Policy

Hadoop Cloudera Installation CM6


Restart failed services

Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6


Hadoop Cloudera Installation CM6