Hadoop单机配置

hadoop单机配置记录

1、配置环境
1)VMware虚拟机，centos7最小化安装

2、centos最小化安装默认不开启网络
   1)开启网络
   命令：vi /etc/sysconfig/network-scripts/ifcfg-ens33 #版本不同最后文件名略有不同
   修改最后一行ONBOOT=yes
   2)重启网络服务
   命令：service network restart
   3)查看ip
   命令：ip add

3、开启ssh服务
   命令：sudo service sshd start

   1)注：如果没有ssh请安装
   yum list installed | grep openssh-server
   参考：https://www.cnblogs.com/kinwing/p/11134179.html

4、配置免密
   1)ssh-keygen -t dsa -P ''
   2)cat ./.ssh/id_dsa.pub >> ./.ssh/authorized_keys

4、关闭防火墙
   1)查看防火墙状态：systemctl status firewalld.service
   2)关闭防火墙：systemctl stop firewalld.service
   3)设置开机不启动：systemctl disable firewalld.service

5、安装JDK配置环境变量
   1)解压java包 tar -zxvf ./OpenJDK11U-jdk_x64_linux_openj9_linuxXL_11.0.10_9_openj9-0.24.0.tar.gz
   2)vi /etc/profile
   3)JAVA_HOME JRE_HOME CLASSPATH PATH 的配置
       export JAVA_HOME=/usr/jdk-11.0.10+9
       export JRE_HOME=/usr/jdk-11.0.10+9
       export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
       export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
   4)编辑完之后，执行命令 source /etc/profile 使配置生效
   5)输入命令：java -version 查看刚刚的JDK配置是否生效

5、安装配置hadoop文件
   1)解压hadoop包:tar -zxvf ./hadoop-3.2.2.tar.gz
   2)添加环境变量:
       export HADOOP_HOME=/usr//usr/hadoop-3.2.2
       export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
       export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
       export PATH=$PATH:$HADOOP_HOME/bin
   3)在root下创建文件夹，（测试环境用的root用户，偷懒了）
       mkdir /root/hadoop
       mkdir /root/hadoop/tmp
       mkdir /root/hadoop/var
       mkdir /root/hadoop/dfs
       mkdir /root/hadoop/dfs/name
       mkdir /root/hadoop/dfs/data
   4)修改 core-site.xml
       <property>
           <name>hadoop.tmp.dir</name>
           <value>/root/hadoop/tmp</value>
           <description>Abase for other temporary directories.</description>
       </property>
           <property>
           <name>fs.default.name</name>
           <value>hdfs://localhost:9000</value>
       </property>
   5)修改 hadoop-env.sh
       执行命令 vi hadoop-env.sh，将${JAVA_HOME} 修改为自己的JDK路径
       export JAVA_HOME=/usr/jdk-11.0.10+9
   6)修改 hdfs-site.xml
       <property>
           <name>dfs.name.dir</name>
           <value>/root/hadoop/dfs/name</value>
           <description>Path on the local filesystem where theNameNode stores the namespace and transactions logs persistently.</description>
       </property>
       <property>
           <name>dfs.data.dir</name>
           <value>/root/hadoop/dfs/data</value>
           <description>Comma separated list of paths on the localfilesystem of a DataNode where it should store its blocks.</description>
       </property>
       <property>
           <name>dfs.replication</name>
           <value>1</value>
       </property>
       <property>
           <name>dfs.permissions</name>
           <value>false</value>
           <description>need not permissions</description>
       </property>
   7)修改 mapred-site.xml
       <property>
           <name>mapred.job.tracker</name>
           <value>localhaost:9001</value>
       </property>
       <property>
           <name>mapred.local.dir</name>
           <value>/root/hadoop/var</value>
       </property>
       <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
       </property>
   8)到此处Hadoop的单机模式配置就完成了

6、启动验证服务
   1)执行命令 ./bin/hadoop namenode -format ，初始化hadoop
   2)修改 ./sbin/start-dfs.sh
       在前面添加如下代码
       HDFS_DATANODE_USER=root
       HADOOP_SECURE_DN_USER=hdfs
       HDFS_NAMENODE_USER=root
       HDFS_SECONDARYNAMENODE_USER=root
   3)修改 ./sbin/start-yarn.sh
       在前面添加如下代码
       YARN_RESOURCEMANAGER_USER=root
       HADOOP_SECURE_DN_USER=yarn
       YARN_NODEMANAGER_USER=root

   4)启动服务
       ./sbin/start-all

   5)执行命令jps查看目前启动的进程
   6)网页验证服务是否清单
       http://localhost:8088/cluster
       http://localhost:9870/

****************************完****************************