Hadoop Eco System Install without Ambari (ㅠㅠ)

Hadoop 3.x 가 아닌 Hadoop 2.x 버전 입니다.

옛날 버전 Hadoop ECO System 구축

Ambari 사용화 되면서... Hadoop-ECO System 설치는... 한땀한땀... 해야한다...

 

설치 할 리스트

1. QuorumPeerMain (Zookeeper)
2. NameNode
3. DataNode
4. NodeManager
5. JournalNode
6. ResourceManager
7. JobHistoryServer
8. DFSZKFailoverController

9. Solr
10. Ranger
11. Hive
12. Hive-Metastore
13. PrestoServer (trino)

 

Name	Version	비고
Hadoop	2.7.3	- Namenode - Datanode - NodeManager - Journalnode - ResourceManager - JobHistoryServer - DFSZKFailoverController
Zookeeper	3.6.3	- QuorumPeerMain
Ranger-admin	1.2.0 & 2.0.0	- Hive 때문에 Ranger 버전을 1.2.0 진행         (/opt/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins           여기서 hive 플러그인 버전 변경) - Presto 를 사용하기 위해 2.0.0 이상 버전 필요
Hive & Hive-Metastore	2.3.9
Presto	315	만약 버전 315 이상은 Ranger-1.2.0 빌드시 Fail (구글 검색 단어 : trino-315)
Spark	3.1.2-2.7	spark 3.1.2 & hadoop 2.7
Solr	6.6.1	6.6.1 이상 버전은 Ranger-1.2.0 빌드시 Fail
Tez	0.9.2

 

Pre-requirements

# 각 VM에 진행 (미리 진행하는걸 권장)

# 필요한 lib
yum install epel-release -y
yum install -y git cmake git-core git-svn subversion checkinstall build-essential dh-make debhelper ant ant-optional autoconf automake liblzo2-dev libzip-dev sharutils libfuse-dev reprepro libtool libssl-dev asciidoc xmlto ssh curl gcc gcc-c++ make fuse protobuf protobuf-devel protobuf-java protobuf-c-compiler protobuf-compiler autoconf automake libtool shareutils asciidoc xmlto lzo-devel zlib-devel fuse-devel openssl-devel python-devel libxml2-devel libxslt-devel cyrus-sasl-devel sqlite-devel mysql-devel openldap-devel rpm-build create-repo redhat-rpm-config wget java-1.8.0-openjdk java-1.8.0-openjdk-devel telnet vim

# check
protoc --version

# create directory
mkdir -p /data/hadoop-eco/namenode
mkdir -p /data/hadoop-eco/datanode
mkdir -p /data/hadoop-eco/journalnode
mkdir -p /data/hadoop-eco/mapred/local
mkdir -p /data/hadoop-eco/mapred/mr-history/done
mkdir -p /data/hadoop-eco/mapred/mr-history/tmp
mkdir -p /data/hadoop-eco/mapred/system
mkdir -p /data/hadoop-eco/yarn/local
mkdir -p /data/hadoop-eco/tmp
mkdir -p /data/hadoop-eco/journalnode
mkdir -p /data/hadoop-eco/yarn
chown -R hadoop:hadoop /data/hadoop-eco/namenode
chown -R hadoop:hadoop /data/hadoop-eco/datanode
chown -R hadoop:hadoop /data/hadoop-eco/journalnode
chown -R hadoop:hadoop /data/hadoop-eco/mapred
chown -R hadoop:hadoop /data/hadoop-eco/yarn
chown -R hadoop:hadoop /data/hadoop-eco/tmp
chown -R hadoop:hadoop /data/hadoop-eco/journalnode

# group 생성
groupadd -g 1001 hadoop

 

설치 순서 중요함

1. zookeeper

https://choonglee.tistory.com/12

Zookeeper 3.6.3 Install

2. hadoop