目錄一、hadoop單機(jī)模式1.1 部署1.2 HDFS創(chuàng)建目錄上傳測(cè)試文件二、完全分布式2.1 server1部署NFS共享服務(wù)2.2 修改Hadoop核心配置文件2.3 server2、server3掛載NFS共享目錄2.4 格式化2.5 HDFS集群測(cè)試創(chuàng)建目錄上傳文件三、集群節(jié)點(diǎn)擴(kuò)容一、hadoop單機(jī)模式1.1 部署官方安裝網(wǎng)址Index of /hadoop/common官方指導(dǎo)文件Apache Hadoop 3.3.5 – Hadoop: Setting up a Single Node Cluster.下載安裝軟件包解壓縮并創(chuàng)建軟連接創(chuàng)建一個(gè)普通用戶(hù)hadoop并登入tar zxf hadoop-3.3.6.tar.gztar zxf jdk1.8.0_181-linux-x64.tar.gzln -s hadoop-3.3.6 hadoopln -s jdk1.8.0_181/ jdk配置Java環(huán)境變量編輯用戶(hù)家目錄.bashrc只對(duì)hadoop這個(gè)用戶(hù)生效hadoop?env.sh指定JAVA_HOME[hadoopserver1 hadoop]$ cd etc[hadoopserver1 etc]$ lshadoop[hadoopserver1 etc]$ cd hadoop[hadoopserver1 hadoop]$ vim hadoop-env.sh把本地xml配置文件復(fù)制到本地input文件夾[hadoopserver1 hadoop]$ cp etc/hadoop/*.xml input運(yùn)行官方grep示例MR程序[hadoopserver1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output dfs[a-z.][hadoopserver1 hadoop]$ cat output/*[hadoopserver1 hadoop]$ vim etc/hadoop/core-site.xmlfs.defaultFSHDFS文件系統(tǒng)默認(rèn)地址客戶(hù)端訪問(wèn)HDFS的入口單機(jī)寫(xiě)localhost:9000[hadoopserver1 hadoop]$ vim etc/hadoop/hdfs-site.xmlhdfs?site.xml副本數(shù)配置免密對(duì)loaclhost這里就不詳細(xì)展示格式化NameNode格式化會(huì)清空HDFS全部數(shù)據(jù)已經(jīng)存數(shù)據(jù)的集群絕對(duì)不能重復(fù)執(zhí)行[hadoopserver1 hadoop]$ bin/hdfs namenode -format生成NameNode元數(shù)據(jù)目錄初始化fsimage鏡像文件建立HDFS文件系統(tǒng)。啟動(dòng)HDFS[hadoopserver1 hadoop]$ sbin/start-dfs.sh9870端口瀏覽器正常訪問(wèn)1.2 HDFS創(chuàng)建目錄上傳測(cè)試文件把本地文件上傳分布式文件系統(tǒng) HDFS給 MapReduce 任務(wù)做輸入數(shù)據(jù)源。[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir input[hadoopserver1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml input先刪除本地舊outpu交mr任務(wù)讀取hdfs input目錄正則匹配dfs[a?z.]輸出到hdfs output[hadoopserver1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output dfs[a-z.][hadoopserver1 hadoop]$ rm -rf output[hadoopserver1 hadoop]$ bin/hdfs dfs -get output output[hadoopserver1 hadoop]$ bin/hdfs dfs -cat output/*二、完全分布式2.1 server1部署NFS共享服務(wù)server1和server2、server3上安裝nfs-utils安裝nfs服務(wù)組件實(shí)現(xiàn)目錄網(wǎng)絡(luò)共享server2、server3可以掛載server1的/home/hadoop。yum install -y nfs-utils[rootserver1 etc]# vim exports[rootserver1 etc]# systemctl enable --now nfs[rootserver1 etc]# showmount -e查看本機(jī)已經(jīng)共享出來(lái)的目錄驗(yàn)證exports配置是否生效cd hadoop/etc/hadoop2.2 修改Hadoop核心配置文件[hadoopserver1 hadoop]$ vim core-site.xmlHDFS 主節(jié)點(diǎn)地址完全分布式必須寫(xiě)NameNode主機(jī)名 server1不能寫(xiě)[localhost]。集群所有 DataNode 通過(guò)這個(gè)地址去找 NameNode。[hadoopserver1 hadoop]$ vim hdfs-site.xml當(dāng)前集群有2臺(tái)DataNodeserver2、server3副本數(shù)設(shè)置2[hadoopserver1 hadoop]$ vim workers填入dataNode主機(jī)名2.3 server2、server3掛載NFS共享目錄[rootserver2 ~]# mount 192.168.159.133:/home/hadoop/ /home/hadoop/2.4 格式化停止舊服務(wù)[hadoopserver1 hadoop]$ sbin/stop-dfs.sh[hadoopserver1 hadoop]$ bin/hdfs namenode -format啟動(dòng)完全分布式 HDFS 集群[hadoopserver1 hadoop]$ sbin/start-dfs.sh2.5 HDFS集群測(cè)試創(chuàng)建目錄上傳文件[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir input[hadoopserver1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml input掛載成功后也會(huì)有server1的文件等三、集群節(jié)點(diǎn)擴(kuò)容原有集群server1 (NameNodeResourceManager)、server2 (DataNodeNodeManager)、server3 (DataNodeNodeManager)擴(kuò)容目標(biāo)新增server4同時(shí)作為DataNodeHDFS 數(shù)據(jù)節(jié)點(diǎn) NodeManagerYARN計(jì)算節(jié)點(diǎn)環(huán)境沿用NFS共享/home/hadoopserver4掛載之后直接復(fù)用jdk、hadoop整套程序不用重新解壓部署hadoop。新開(kāi)一臺(tái)虛擬機(jī)server4初始化新節(jié)點(diǎn) server4[rootserver4 ~]# useradd hadoop[rootserver4 ~]# passwd hadoop[rootserver4 ~]# mount 192.168.159.133:/home/hadoop/ /home/hadoop/server1修改MapReduce配置mapred?site.xml[hadoopserver1 hadoop]$ vim etc/hadoop/mapred-site.xmlserver1修改MapReduce配置yarn?site.xml[hadoopserver1 hadoop]$ vim etc/hadoop/yarn-site.xmlserver4手動(dòng)后臺(tái)啟動(dòng)DataNode、NodeManager[rootserver4 ~]# su - hadoop[hadoopserver4 ~]$ hdfs --daemon start datanode[hadoopserver4 ~]$ yarn --daemon start nodemanagerserver1更新workers文件啟動(dòng)YARN集群查看集群節(jié)點(diǎn)狀態(tài)[hadoopserver1 hadoop]$ vim workers[hadoopserver1 hadoop]$ sbin/start-yarn.sh[hadoopserver1 hadoop]$ hdfs dfsadmin -report[hadoopserver1 hadoop]$ vim etc/hadoop/yarn-site.xml新加入兩行內(nèi)容server2也需要啟動(dòng)[hadoopserver3 ~]$ yarn --daemon start nodemanager添加成功瀏覽器訪問(wèn)192.168.159.133:8088端口也就是server1