原Centos安装ApacheHadoop2.7.7

Centos安装ApacheHadoop2.7.7

准备文件:
hadoop-2.7.7.tar.gz

一、安装JDK

安装Hadoop之前需要保证JDK8成功安装
java -version

二、上传Hadoop压缩包

上传成功后解压文件：tar -zxvf hadoop-2.7.7.tar.gz
假设解压后的Hadoop目录为: /home/hadoop/hadoop-2.7.7

注意配置HADOOP_HOME环境变量
export HADOOP_HOME=/home/hadoop/hadoop-2.7.7
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

三、配置服务器免密登录

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

执行完成后: ssh master命令则能够跳过输入密码阶段直接连接

四、修改配置

==配置文件所在位置: $HADOOP_HOME/etc/hadoop==

1. core-size.xml

   <configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
   </configuration>

2. hdfs-size.xml

   <configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
   </configuration>

3. mapred-site.xml

   <configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
   </configuration>

4. yarn-site.xml

   <configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
   </configuration>

   五、启动

5. 初始化HDFS hdfs namenode -format
6. 启动HDFS start-dfs.sh
7. 启动YARN start-yarn.sh
8. 启动成功后访问http://master:50070查看HDFS状态

六、相关命令

start-dfs.sh 启动HDFS(namenode/datanode/SecondaryNamenode)
stop-dfs.sh 停止HDFS
start-yarn.sh 启动Yarn
stop-yarn.sh 停止Yarn
start-all.sh 启动HDFS和Yarn
stop-all.sh 停止HDFS和Yarn