正文

CentOS 7安装Hadoop集群

IP云
IP云V管理员/7 阅读
11/30
提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

准备三台虚拟机，ip分别为192.168.220.10（master）、192.168.220.11（slave1）、192.168.220.12（slave2）

准备好jdk-6u45-linux-x64.bin和hadoop-1.2.1-bin.tar.gz，放在/usr/local/src/目录下

安装JDK （每台虚拟机都安装）

1.进入到/usr/local/src/目录，执行./jdk-6u45-linux-x64.bin

2.修改~/.bashrc，在文件末尾增加三行 

export JAVA_HOME=/usr/local/src/jdk1.6.0_45

export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin

3.使环境变量生效，执行source ~/.bashrc

安装Hadoop

在192.168.220.10机器上安装hadoop

1.进入到/usr/local/src/目录，解压hadoop-1.2.1-bin.tar.gz，执行tar -zxf hadoop-1.2.1-bin.tar.gz

2.修改配置文件

masters文件 

master

slaves文件 

slave1

slave2

core-site.xml文件 

<configuration>

	<property>

		<name>hadoop.tmp.dir</name>

		<value>/usr/local/src/hadoop-1.2.1/tmp</value>

	</property>

	<property>

		<name>fs.default.name</name>

		<value>hdfs://192.168.220.10:9000</value>

	</property>

</configuration>

mapred-site.xml文件 

<configuration>

	<property>

		<name>mapred.job.tracker</name>

		<value>http://192.168.220.10:9001</value>

	</property>

</configuration>

hdfs-site.xml文件 

<configuration>

	<property>

		<name>dfs.replication</name>

		<value>3</value>

	</property>

</configuration>

hadoop-env.sh文件，在后面添加一行 

export JAVA_HOME=/usr/local/src/jdk1.6.0_45

3.将/usr/local/src/hadoop-1.2.1目录拷贝到192.168.220.11、192.168.220.12机器上

配置hostname

配置192.168.220.10的主机名为master

1.执行hostname master

2.修改/etc/hostname文件 

master

修改192.168.220.11的主机名为slave1，修改192.168.220.12的主机名为slave2

配置host文件

三台机器的/etc/hosts文件末尾添加以下代码 

192.168.220.10    master

192.168.220.11    slave1

192.168.220.12    slave2

关闭防火墙和SELinux

分别在每台机器上执行 

systemctl stop firewalld.service

setenforce

配置SSH

1.在192.168.220.10上执行ssh-keygen，在~目录下新增.ssh目录，目录中的文件为id_rsa，id_rsa.pub

2.将id_rsa.pub拷贝为authorized_keys 

cp id_rsa.pub authorized_keys

3.在192.168.220.11和192.168.220.12上分别执行ssh-keygen

4.将192.168.220.11和192.168.220.12上id_rsa.pub的内容分别拷贝到192.168.220.10的authorized_keys文件中，如下： 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9mGRhFOdcoHw9GUnKQmqThNKpsyah93Dtq/d8RICGWIHDRJ3GXd0sEcb743ejwbuCMmtlhheXcU0FuyA6Cm0jvMyvDfaPKArtxl6KT7Z93uC0VDCXDRomueux81HAIVjc7ZqlXwVeYs1LITxEeJykKlFOXvK7JexWhWGdMMADwxbFMbaNsZ9EwRxcFLFtNg65FQ+u8CIV9KR3D02kemwLCsP+xiRcgs+wirQPm5JM+2cJoLsVQBz3Hk335IsEhc1Xb9Cralo8Tt8gh/ho8K/1pVjvyW1b0LkP9HGNdwVYD9wkWdEJRkryLXBEXpjk4xu+riF+N4rOzJD root@master

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn79fdfR/NjzPVD3NPj1vBBfQdVOrv7jeb4UJCOsd7xioPRiz8gOQnOmhu5C+GchbyGA+tg5pXwnNJTOO2wn32U4lOPndW0okN/wqyN4vgq/taJi7JgY/8rneBiGaIIdNIy/pAGlMwb53Qn766adetMhsxYMD2l4uxmbVVjzCRb8QP5EsAYTmmFOODzJsPm70uF3j1Q8zGavYg0wFSYR/yECQns4DBSuBJNxdGY6PskBXqurahwi5yaR3vWV1Ix4wtB6BYuQomEnGdzOSfrBMZ/yc5tXo0xmEfY7wFkize6z9Pm2E3oDoMR18YkwT1Cz6fHikVILA9cldtL root@slave1

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCydYCvASzCZggks4hMqOcYSGLO2eAvocWezNOMwspTfpJ105Jumb/vf5h6cRZeckq56IvhSV6t6mytk4pZoZjjZPSmWvCwLtMRMPShNbA3BYtj5V3WRKV8ZcMrNdD//U7iHHoJm57vI/m+XO42YSYjPw7JDkb8Ij9b6zgI3fyvbSSYeXb451PlyJLHdxIzRMAaZDSbAML9e7EO8VJB9Wf9bXpow4+VeP33it3kgMNUlHQtyqduSwYGxVVtGsUTJkxnuRsbWeeA1/pp8MNFKUgBTMALTVHByglgZqwGcbblJxsG832PIZNRECIFqorm6odftjnT4DR7/0yR root@slave2

5.将192.168.220.10的authorized_keys文件拷贝到192.168.220.11、192.168.220.12机器上

配置完成后三台机器可以互相访问不用密码

好了，到这里Hadoop集群就配置完了，让我们来使用下吧

在192.168.220.10上格式化namenode

进入到/usr/local/src/hadoop-1.2.1/bin目录，执行 

./hadoop namenode -format

出现 

// :: INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = master/192.168.220.10

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.2.1

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG:   java = 1.6.0_45

************************************************************/

// :: INFO util.GSet: Computing capacity for map BlocksMap

// :: INFO util.GSet: VM type       = -bit

// :: INFO util.GSet: 2.0% max memory =

// :: INFO util.GSet: capacity      = ^ =  entries

// :: INFO util.GSet: recommended=, actual=

// :: INFO namenode.FSNamesystem: fsOwner=root

// :: INFO namenode.FSNamesystem: supergroup=supergroup

// :: INFO namenode.FSNamesystem: isPermissionEnabled=true

// :: INFO namenode.FSNamesystem: dfs.block.invalidate.limit=

// :: INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval= min(s), accessTokenLifetime= min(s)

// :: INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length =

// :: INFO namenode.NameNode: Caching file names occuring more than  times

// :: INFO common.Storage: Image file /usr/local/src/hadoop-1.2./tmp/dfs/name/current/fsimage of size  bytes saved in  seconds.

// :: INFO namenode.FSEditLog: closing edit log: position=, editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits

// :: INFO namenode.FSEditLog: close success: truncate to , editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits

// :: INFO common.Storage: Storage directory /usr/local/src/hadoop-1.2./tmp/dfs/name has been successfully formatted.

// :: INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at master/192.168.220.10

************************************************************/

用jps查看进程 

[root@master bin]# jps

 JobTracker

 NameNode

 SecondaryNameNode

 Jps

在192.168.220.11查看进程 

 DataNode

 Jps

 TaskTracker

在192.168.220.12查看进程 

 DataNode

 TaskTracker

 Jps

测试下：

执行./hadoop fs -ls / 

drwxr-xr-x   - root supergroup           -- : /usr

上传文件，执行./hadoop fs -put /root/w.txt /

查看文件执行./hadoop fs -cat /w.txt，显示 

ddd

成功

注：

问题：上传文件时出现以下错误 

[root@master bin]# ./hadoop fs -put /root/a.txt /

// :: WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to  nodes, instead of

    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)

    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

    at org.apache.hadoop.ipc.Client.call(Client.java:)

    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)

    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)

    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)

    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:)

// :: WARN hdfs.DFSClient: Error Recovery for null bad datanode[] nodes == null

// :: WARN hdfs.DFSClient: Could not get block locations. Source file "/a.txt" - Aborting...

put: java.io.IOException: File /a.txt could only be replicated to  nodes, instead of

// :: ERROR hdfs.DFSClient: Failed to close file /a.txt

org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to  nodes, instead of

    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)

    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)

    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

    at org.apache.hadoop.ipc.Client.call(Client.java:)

    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)

    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)

    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)

    at com.sun.proxy.$Proxy1.addBlock(Unknown Source)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)

    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:)

解决方法：关闭防火墙

相关阅读

目录[+]