´óÊý¾ÝÈëÃÅÖ®Hadoop»ù´¡Ñ§Ï°´óÊý¾ÝÓ¦ÓÃ
Ò»¡¢Êý¾ÝµÄ´æ´¢£º·Ö²¼Ê½Îļþϵͳ(·Ö²¼Ê½´æ´¢)
¶þ¡¢Êý¾ÝµÄ¼ÆË㣺·Ö²¿Êð¼ÆËã
»ù´¡ÖªÊ¶
ѧϰ´óÊý¾ÝÐèÒª¾ß±¸Java֪ʶ»ù´¡¼°Linux֪ʶ»ù´¡
ѧϰ·Ïß
(1)Java»ù´¡ºÍLinux»ù´¡
(2)HadoopµÄѧϰ£ºÌåϵ½á¹¹¡¢ÔÀí¡¢±à³Ì
µÚÒ»½×¶Î£ºHDFS¡¢MapReduce¡¢HBase(NoSQLÊý¾Ý¿â)
µÚ¶þ½×¶Î£ºÊý¾Ý·ÖÎöÒýÇæ -> Hive¡¢Pig
Êý¾Ý²É¼¯ÒýÇæ -> Sqoop¡¢Flume
µÚÈý½×¶Î£ºHUE£ºWeb¹ÜÀí¹¤¾ß
ZooKeeper£ºÊµÏÖHadoopµÄHA
Oozie£º¹¤×÷Á÷ÒýÇæ
(3)SparkµÄѧϰ
µÚÒ»½×¶Î£ºScala±à³ÌÓïÑÔ
µÚ¶þ½×¶Î£ºSpark Core -> »ùÓÚÄÚ´æ¡¢Êý¾ÝµÄ¼ÆËã
µÚÈý½×¶Î£ºSpark SQL -> ÀàËÆÓÚmysql µÄsqlÓï¾ä
µÚËĽ׶ΣºSpark Streaming ->½øÐÐÁ÷ʽ¼ÆË㣺±ÈÈ磺×ÔÀ´Ë®³§
(4)Apache Storm ÀàËÆ£ºSpark Streaming ->½øÐÐÁ÷ʽ¼ÆËã
NoSQL£ºRedis»ùÓÚÄÚ´æµÄÊý¾Ý¿â
HDFS
·Ö²¼Ê½Îļþϵͳ ½â¾öÒÔÏÂÎÊÌ⣺
Ó²Å̲»¹»´ó£º¶à¼¸¿éÓ²ÅÌ£¬ÀíÂÛÉÏ¿ÉÒÔÎÞÏÞ´ó
Êý¾Ý²»¹»°²È«£ºÈßÓà¶È£¬hdfsĬÈÏÈßÓàΪ3 £¬ÓÃˮƽ¸´ÖÆÌá¸ßЧÂÊ£¬´«Êä°´ÕÕÊý¾Ý¿âΪµ¥Î»£ºHadoop1.x 64M£¬Hadoop2.x 128M
¹ÜÀíÔ±£ºNameNode Ó²ÅÌ£ºDataNode
![image.png]()
MapReduce
»ù´¡±à³ÌÄ£ÐÍ£º°ÑÒ»¸ö´óÈÎÎñ²ð·Ö³ÉСÈÎÎñ£¬ÔÙ½øÐлã×Ü
MRÈÎÎñ£ºJob = Map + Reduce
MapµÄÊä³öÊÇReduceµÄÊäÈë¡¢MRµÄÊäÈëºÍÊä³ö¶¼ÊÇÔÚHDFS
MapReduceÊý¾ÝÁ÷³Ì·ÖÎö£º
MapµÄÊä³öÊÇReduceµÄÊäÈ룬ReduceµÄÊäÈëÊÇMapµÄ¼¯ºÏ
HBase
ʲôÊÇBigTable?: °ÑËùÓеÄÊý¾Ý±£´æµ½Ò»ÕűíÖУ¬²ÉÓÃÈßÓà ---> ºÃ´¦£ºÌá¸ßЧÂÊ
ÒòΪÓÐÁËbigtableµÄ˼Ï룺NoSQL£ºHBaseÊý¾Ý¿â
HBase»ùÓÚHadoopµÄHDFSµÄ
ÃèÊöHBaseµÄ±í½á¹¹
ºËÐÄ˼ÏëÊÇ£ºÀûÓÿռ任ЧÂÊ
Hadoop»·¾³´î½¨
»·¾³×¼±¸
Linux»·¾³¡¢JDK¡¢
°²×°
1¡¢°²×°jdk¡¢²¢ÅäÖû·¾³±äÁ¿
vim /etc/profile ĩβÌí¼Ó
![image.png]()
2¡¢½âѹhadoop-3.0.0.tar.gz¡¢²¢ÅäÖû·¾³±äÁ¿
tar -zxvf hadoop-3.0.0.tar.gz -C /usr/local/
mv hadoop-3.0.0/ hadoop
vim /etc/profile ĩβÌí¼Ó
ÅäÖÃ
HadoopÓÐÈýÖÖ°²×°Ä£Ê½£º
±¾µØÄ£Ê½ £º
1̨Ö÷»ú
²»¾ß±¸HDFS£¬Ö»ÄܲâÊÔMapReduce³ÌÐò
α·Ö²¼Ä£Ê½£º
1̨Ö÷»ú
¾ß±¸HadoopµÄËùÓй¦ÄÜ£¬ÔÚµ¥»úÉÏÄ£ÄâÒ»¸ö·Ö²¼Ê½µÄ»·¾³
(1)HDFS£ºÖ÷£ºNameNode£¬Êý¾Ý½Úµã£ºDataNode
(2)Yarn£ºÈÝÆ÷£¬ÔËÐÐMapReduce³ÌÐò
Ö÷½Úµã£ºResourceManager
´Ó½Úµã£ºNodeManager
È«·Ö²¼Ä£Ê½£º
ÖÁÉÙ3̨
ÎÒÃÇÒÔα·Ö²¼Ä£Ê½ÎªÀýÅäÖãº
ÐÞ¸Ähdfs-site.xml£ºÈßÓà¶È1¡¢È¨ÏÞ¼ì²éfalse
<!--ÅäÖÃÈßÓà¶ÈΪ1-->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!--ÅäÖÃȨÏÞ¼ì²éΪfalse-->
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
ÐÞ¸Äcore-site.xml
<!--ÅäÖÃHDFSµÄNameNode-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.56.102:9000</value>
</property>
<!--ÅäÖÃDataNode±£´æÊý¾ÝµÄλÖÃ-->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
ÐÞ¸Ämapred-site.xml
<!--ÅäÖÃMRÔËÐеĿò¼Ü-->
<property>
<name>mapreduce.framework.name</name>
<value>yar</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>
/usr/local/hadoop/etc/hadoop,
/usr/local/hadoop/share/hadoop/common/*,
/usr/local/hadoop/share/hadoop/common/lib/*,
/usr/local/hadoop/share/hadoop/hdfs/*,
/usr/local/hadoop/share/hadoop/hdfs/lib/*,
/usr/local/hadoop/share/hadoop/mapreduce/*,
/usr/local/hadoop/share/hadoop/mapreduce/lib/*,
/usr/local/hadoop/share/hadoop/yarn/*,
/usr/local/hadoop/share/hadoop/yarn/lib/*,
</value>
</property>
ÐÞ¸Äyarn-site.xml
<!--ÅäÖÃResourceManagerµØÖ·-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>192.168.56.102</value>
</property>
<!--ÅäÖÃNodeManagerÖ´ÐÐÈÎÎñµÄ·½Ê½-->
<property>
<name>yarn.nodemanager.aux-service</name>
<value>mapreduce_shuffle</value>
</property>
¸ñʽ»¯NameNode
hdfs namenode -format
¿´µ½common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted±íʾ¸ñʽ»¯³É¹¦
Æô¶¯
start-all.sh
(*)HDFS£º´æ´¢Êý¾Ý
(*)YARN£º
·ÃÎÊ
£¨*£©ÃüÁîÐÐ
£¨*£©Java Api
£¨*£©WEB Console
HDFS: :50070
Yarn: :8088


