Hadoop生态圈-HUE环境部署

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

Hadoop生态圈-HUE环境部署

作者：尹正杰

一.HUE简介

1>.HUE的由来

HUE全称是HadoopUser Experience，看这名字就知道怎么回事了吧。没错，直白来说就是Hadoop用户体验，是一个开源的Apache Hadoop UI系统，由Cloudera Desktop演化而来，最后Cloudera公司将其贡献给Apache基金会的Hadoop社区，它是基于Python Web框架Django实现的。通过使用HUE我们可以在浏览器端的Web控制台上与Hadoop集群进行交互来分析处理数据。

2>.官网地址

HUE官网地址可参考：http://gethue.com/overview/。

二.HUE安装

1>.部署环境介绍（我使用的是CentOS 7.2的操作系统）

2>.安装开发JDK版本

现在你生成环境用的Java开发版本，关于Java的下载地址可参考官网：http://www.oracle.com/technetwork/java/javase/downloads，下载后通过上传工具（如winscp等工具，或者你也可以使用linux自带的rz命令进行上传）

[yinzhengjie@s100 ~]$ sudo yum -y install lrzsz
[sudo] password for yinzhengjie:
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.huaweicloud.com
* extras: mirrors.huaweicloud.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package lrzsz.x86_64 :0.12.-.el7 will be installed
--> Finished Dependency ResolutionDependencies Resolved============================================================================================================
Package Arch Version Repository Size
============================================================================================================
Installing:
lrzsz x86_64 0.12.-.el7 base kTransaction Summary
============================================================================================================
Install PackageTotal download size: k
Installed size: k
Downloading packages:
lrzsz-0.12.-.el7.x86_64.rpm | kB ::
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : lrzsz-0.12.-.el7.x86_64 /
Verifying : lrzsz-0.12.-.el7.x86_64 / Installed:
lrzsz.x86_64 :0.12.-.el7 Complete!
[yinzhengjie@s100 ~]$

安装linux自带的上传工具命令（[yinzhengjie@s100 ~]$ sudo yum -y install lrzsz）

[yinzhengjie@s100 ~]$ ll
total
[yinzhengjie@s100 ~]$ rz #会弹出对话框，选择你要上传的文件即可[yinzhengjie@s100 ~]$ ll
total
-rw-r--r--. yinzhengjie yinzhengjie Aug : jdk-8u181-linux-x64.tar.gz
[yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ sudo mkdir /soft
[yinzhengjie@s100 ~]$ sudo chown yinzhengjie:yinzhengjie /soft/
[yinzhengjie@s100 ~]$ tar zxf jdk-8u181-linux-x64.tar.gz -C /soft/
[yinzhengjie@s100 ~]$ ln -s /soft/jdk1..0_181/ /soft/jdk
[yinzhengjie@s100 ~]$ ll /soft/
total
lrwxrwxrwx. yinzhengjie yinzhengjie Aug : jdk -> /soft/jdk1..0_181/
drwxr-xr-x. yinzhengjie yinzhengjie Jul : jdk1..0_181
[yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ sudo vi /etc/profile
[yinzhengjie@s100 ~]$ more /etc/profile | tail - #添加java的环境变量
#ADD JAVA_PATH
JAVA_HOME=/soft/jdk
PATH=$PATH:$JAVA_HOME/bin
[yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ source /etc/profile #重新加载系统配置文件，使得配置文件在当前shell下生效
[yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ java -version #查看java版本
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK -Bit Server VM (build 25.181-b13, mixed mode)
[yinzhengjie@s100 ~]$

3>.准备环境依赖

查看官网安装文档（http://archive.cloudera.com/cdh5/cdh/5/hue-3.7.0-cdh5.3.0/manual.html），我们就可以看到相关的依赖环境如下：（CentOS具体依赖包参考：http://cloudera.github.io/hue/latest/admin-manual/manual.html#installation）

Centos安装操作系统环境如下:

[yinzhengjie@s100 ~]$sudo yum -y install apache-maven ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel
[sudo] password for yinzhengjie:
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.huaweicloud.com
* extras: mirrors.huaweicloud.com
* updates: mirrors.aliyun.com
..........接下来的信息我就省略了，安装过程以及成功后会有以下输出：（见下图）

安装过程如下：

安装成功如下：

4>.下载HUE安装包

[yinzhengjie@s100 ~]$ sudo yum -y install wget
[sudo] password for yinzhengjie:
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.huaweicloud.com
* extras: mirrors.huaweicloud.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 :1.14-.el7_4. will be installed
--> Finished Dependency ResolutionDependencies Resolved============================================================================================================
Package Arch Version Repository Size
============================================================================================================
Installing:
wget x86_64 1.14-.el7_4. base kTransaction Summary
============================================================================================================
Install PackageTotal download size: k
Installed size: 2.0 M
Downloading packages:
wget-1.14-.el7_4..x86_64.rpm | kB ::
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : wget-1.14-.el7_4..x86_64 /
Verifying : wget-1.14-.el7_4..x86_64 / Installed:
wget.x86_64 :1.14-.el7_4. Complete!
[yinzhengjie@s100 ~]$

安装下载命令wget（[yinzhengjie@s100 ~]$ sudo yum -y install wget）

5>.删除安装后的openJDK

温馨提示：使用yum安装这些包的同时，也会自动安装openJDK的依赖，所以，请自行删除安装后的openJDK

[yinzhengjie@s100 ~]$ rpm -qa | grep java
javapackages-tools-3.4.-.el7.noarch
java-1.8.-openjdk-headless-1.8.0.181-.b13.el7_5.x86_64
python-javapackages-3.4.-.el7.noarch
tzdata-java-2018e-.el7.noarch
java-1.8.-openjdk-1.8.0.181-.b13.el7_5.x86_64
java-1.8.-openjdk-devel-1.8.0.181-.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps python-javapackages-3.4.-.el7.noarch
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps javapackages-tools-3.4.-.el7.noarch
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.-openjdk-1.8.0.181-.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.-openjdk-devel-1.8.0.181-.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.-openjdk-headless-1.8.0.181-.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$ rpm -qa | grep java
tzdata-java-2018e-.el7.noarch
[yinzhengjie@s100 ~]$

6>.解压HUE安装包

[yinzhengjie@s100 ~]$ ll
total
-rw-r--r--. yinzhengjie yinzhengjie Sep hue-3.7.-cdh5.3.6.tar.gz
-rw-r--r--. yinzhengjie yinzhengjie Aug : jdk-8u181-linux-x64.tar.gz
[yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ tar -zxf hue-3.7.-cdh5.3.6.tar.gz -C /soft/
[yinzhengjie@s100 ~]$ ll /soft/
total
drwxr-xr-x. yinzhengjie yinzhengjie Jul hue-3.7.-cdh5.3.6
lrwxrwxrwx. yinzhengjie yinzhengjie Aug : jdk -> /soft/jdk1..0_181/
drwxr-xr-x. yinzhengjie yinzhengjie Jul : jdk1..0_181
[yinzhengjie@s100 ~]$

7>.编译HUE

[yinzhengjie@s100 ~]$ ll /soft/
total
drwxr-xr-x. yinzhengjie yinzhengjie Jul hue-3.7.-cdh5.3.6
lrwxrwxrwx. yinzhengjie yinzhengjie Aug : jdk -> /soft/jdk1..0_181/
drwxr-xr-x. yinzhengjie yinzhengjie Jul : jdk1..0_181
[yinzhengjie@s100 ~]$ cd /soft/hue-3.7.-cdh5.3.6/
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ ll
total
drwxr-xr-x. yinzhengjie yinzhengjie Jul apps
drwxr-xr-x. yinzhengjie yinzhengjie Jul cloudera
drwxr-xr-x. yinzhengjie yinzhengjie Jul desktop
drwxr-xr-x. yinzhengjie yinzhengjie Jul docs
drwxr-xr-x. yinzhengjie yinzhengjie Jul ext
-rw-r--r--. yinzhengjie yinzhengjie Jul LICENSE.txt
-rw-r--r--. yinzhengjie yinzhengjie Jul Makefile
-rw-r--r--. yinzhengjie yinzhengjie Jul Makefile.sdk
-rw-r--r--. yinzhengjie yinzhengjie Jul Makefile.vars
-rw-r--r--. yinzhengjie yinzhengjie Jul Makefile.vars.priv
drwxr-xr-x. yinzhengjie yinzhengjie Jul maven
-rw-r--r--. yinzhengjie yinzhengjie Jul NOTICE.txt
-rw-r--r--. yinzhengjie yinzhengjie Jul README
drwxr-xr-x. yinzhengjie yinzhengjie Jul tools
-rw-r--r--. yinzhengjie yinzhengjie Jul VERSION
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ make apps#开始编译HUE
--- Creating virtual environment at /soft/hue-3.7.-cdh5.3.6/build/env
python2. /soft/hue-3.7.-cdh5.3.6/tools/virtual-bootstrap/virtual-bootstrap.py \
-qq --no-site-packages /soft/hue-3.7.-cdh5.3.6/build/env
--- /soft/hue-3.7.-cdh5.3.6/build/env ready
make[]: Entering directory `/soft/hue-3.7.-cdh5.3.6/desktop'
make -C core env-install
make[]: Entering directory `/soft/hue-3.7.-cdh5.3.6/desktop/core'
--- Building egg for markdown
running bdist_egg
running egg_info
......... #编译时会出现下图的报错信息

解决方案：

解决方案：
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ sudo vi /usr/include/openssl/x509.h
[sudo] password for yinzhengjie: #删除以下两行：
X509_REVOKED *X509_REVOKED_dup(X509_REVOKED *rev);
X509_REQ *X509_REQ_dup(X509_REQ *req);
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$

使用root权限删除图中的751和752行代码，然后在重新编译即可。编译成功截图如下：

8>.配置HUE

[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ vi desktop/conf/hue.ini
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ grep secret_key= desktop/conf/hue.ini
secret_key=jFE93j;[-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mn<qW5o
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ grep http_host= desktop/conf/hue.ini
http_host=172.30.1.100
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ grep http_port= desktop/conf/hue.ini
http_port=
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$ grep time_zone= desktop/conf/hue.ini
time_zone=Asia/Shanghai
[yinzhengjie@s100 hue-3.7.-cdh5.3.6]$

具体配置如下：

9>.启动HUE

10>.关闭服务器防火墙以及selinux服务。

[yinzhengjie@s100 ~]$ systemctl stop firewalld #临时关闭防火墙
[yinzhengjie@s100 ~]$ systemctl disable firewalld #禁用防火墙的服务开机自启
[yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ su root
Password:
[root@s100 yinzhengjie]# cat /etc/selinux/config # This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted [root@s100 yinzhengjie]#
[root@s100 yinzhengjie]# getenforce
Enforcing
[root@s100 yinzhengjie]#
[root@s100 yinzhengjie]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config
[root@s100 yinzhengjie]#
[root@s100 yinzhengjie]# cat /etc/selinux/config | grep SELINUX= | grep -v ^#
SELINUX=disabled
[root@s100 yinzhengjie]#

11>.访问HUE的webUI并设置管理员账号

Duang~如果你出现以下界面说明你HUE启动成功啦！

三.通过git方式安装HUE

经查阅资料，发现github上也有教程教你如何安装HUE，需要你本地服务器安装git命令，详情请参考：https://github.com/cloudera/hue#development-prerequisites

[yinzhengjie@s100 ~]$ sudo yum -y install git
[sudo] password for yinzhengjie:
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.huaweicloud.com
* extras: mirrors.huaweicloud.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 :1.8.3.1-.el7_5 will be installed
--> Processing Dependency: perl-Git = 1.8.3.1-.el7_5 for package: git-1.8.3.1-.el7_5.x86_64
--> Processing Dependency: rsync for package: git-1.8.3.1-.el7_5.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-1.8.3.1-.el7_5.x86_64
--> Processing Dependency: perl(Git) for package: git-1.8.3.1-.el7_5.x86_64
--> Processing Dependency: perl(Error) for package: git-1.8.3.1-.el7_5.x86_64
--> Processing Dependency: libgnome-keyring.so.()(64bit) for package: git-1.8.3.1-.el7_5.x86_64
--> Running transaction check
---> Package libgnome-keyring.x86_64 :3.12.-.el7 will be installed
---> Package perl-Error.noarch :0.17020-.el7 will be installed
---> Package perl-Git.noarch :1.8.3.1-.el7_5 will be installed
---> Package perl-TermReadKey.x86_64 :2.30-.el7 will be installed
---> Package rsync.x86_64 :3.1.-.el7 will be installed
--> Finished Dependency ResolutionDependencies Resolved==========================================================================================================================================================
Package Arch Version Repository Size
==========================================================================================================================================================
Installing:
git x86_64 1.8.3.1-.el7_5 updates 4.4 M
Installing for dependencies:
libgnome-keyring x86_64 3.12.-.el7 base k
perl-Error noarch :0.17020-.el7 base k
perl-Git noarch 1.8.3.1-.el7_5 updates k
perl-TermReadKey x86_64 2.30-.el7 base k
rsync x86_64 3.1.-.el7 base kTransaction Summary
==========================================================================================================================================================
Install Package (+ Dependent packages)Total download size: 5.0 M
Installed size: M
Downloading packages:
(/): perl-TermReadKey-2.30-.el7.x86_64.rpm | kB ::
(/): libgnome-keyring-3.12.-.el7.x86_64.rpm | kB ::
(/): perl-Error-0.17020-.el7.noarch.rpm | kB ::
(/): perl-Git-1.8.3.1-.el7_5.noarch.rpm | kB ::
(/): rsync-3.1.-.el7.x86_64.rpm | kB ::
(/): git-1.8.3.1-.el7_5.x86_64.rpm | 4.4 MB ::
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.7 MB/s | 5.0 MB ::
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found pre-existing rpmdb problem(s), 'yum check' output follows:
ant-1.9.-.el7.noarch has missing requires of java-devel >= ('', '1.5.0', None)
ant-1.9.-.el7.noarch has missing requires of jpackage-utils
ant-1.9.-.el7.noarch has missing requires of jpackage-utils >= ('', '1.7.5', None)
xalan-j2-2.7.-.el7.noarch has missing requires of jpackage-utils
xerces-j2-2.11.-.el7_0.noarch has missing requires of java
xerces-j2-2.11.-.el7_0.noarch has missing requires of jpackage-utils
xerces-j2-2.11.-.el7_0.noarch has missing requires of jpackage-utils
xml-commons-apis-1.4.-.el7.noarch has missing requires of java
xml-commons-apis-1.4.-.el7.noarch has missing requires of jpackage-utils
xml-commons-apis-1.4.-.el7.noarch has missing requires of jpackage-utils
xml-commons-resolver-1.2-.el7.noarch has missing requires of jpackage-utils
xml-commons-resolver-1.2-.el7.noarch has missing requires of jpackage-utils
Installing : :perl-Error-0.17020-.el7.noarch /
Installing : rsync-3.1.-.el7.x86_64 /
Installing : perl-TermReadKey-2.30-.el7.x86_64 /
Installing : libgnome-keyring-3.12.-.el7.x86_64 /
Installing : perl-Git-1.8.3.1-.el7_5.noarch /
Installing : git-1.8.3.1-.el7_5.x86_64 /
Verifying : git-1.8.3.1-.el7_5.x86_64 /
Verifying : libgnome-keyring-3.12.-.el7.x86_64 /
Verifying : perl-TermReadKey-2.30-.el7.x86_64 /
Verifying : :perl-Error-0.17020-.el7.noarch /
Verifying : rsync-3.1.-.el7.x86_64 /
Verifying : perl-Git-1.8.3.1-.el7_5.noarch / Installed:
git.x86_64 :1.8.3.1-.el7_5 Dependency Installed:
libgnome-keyring.x86_64 :3.12.-.el7 perl-Error.noarch :0.17020-.el7 perl-Git.noarch :1.8.3.1-.el7_5 perl-TermReadKey.x86_64 :2.30-.el7
rsync.x86_64 :3.1.-.el7 Complete!
[yinzhengjie@s100 ~]$