´óÊý¾ÝÈëÃÅÖ®Hadoop»ù´¡Ñ§Ï°´óÊý¾ÝÓ¦ÓÃ

À´Ô´£º»¥ÁªÍø / ×÷ÕߣºSKY / 2018-07-11 18:38 / µã»÷£º
ĿǰÈ˹¤ÖÇÄܺʹóÊý¾Ý»ðÈÈ£¬Ê¹Óõij¡¾°Ò²Ô½À´Ô½¹ã£¬ÈÕ³£¿ª·¢ÖÐǰ¶ËͬѧҲÖð½¥½Ó´¥Á˸ü¶àÓë´óÊý¾ÝÏà¹ØµÄ¿ª·¢ÐèÇó¡£Òò´Ë¶Ô´óÊý¾Ý֪ʶҲÓбØÒª½øÐÐһЩѧϰÀí½â¡£

Ò»¡¢Êý¾ÝµÄ´æ´¢£º·Ö²¼Ê½Îļþϵͳ(·Ö²¼Ê½´æ´¢)

¶þ¡¢Êý¾ÝµÄ¼ÆË㣺·Ö²¿Êð¼ÆËã

»ù´¡ÖªÊ¶

ѧϰ´óÊý¾ÝÐèÒª¾ß±¸Java֪ʶ»ù´¡¼°Linux֪ʶ»ù´¡

ѧϰ·Ïß

(1)Java»ù´¡ºÍLinux»ù´¡

(2)HadoopµÄѧϰ£ºÌåϵ½á¹¹¡¢Ô­Àí¡¢±à³Ì

µÚÒ»½×¶Î£ºHDFS¡¢MapReduce¡¢HBase(NoSQLÊý¾Ý¿â)

µÚ¶þ½×¶Î£ºÊý¾Ý·ÖÎöÒýÇæ -> Hive¡¢Pig

Êý¾Ý²É¼¯ÒýÇæ -> Sqoop¡¢Flume

µÚÈý½×¶Î£ºHUE£ºWeb¹ÜÀí¹¤¾ß

ZooKeeper£ºÊµÏÖHadoopµÄHA

Oozie£º¹¤×÷Á÷ÒýÇæ

(3)SparkµÄѧϰ

µÚÒ»½×¶Î£ºScala±à³ÌÓïÑÔ

µÚ¶þ½×¶Î£ºSpark Core -> »ùÓÚÄÚ´æ¡¢Êý¾ÝµÄ¼ÆËã

µÚÈý½×¶Î£ºSpark SQL -> ÀàËÆÓÚmysql µÄsqlÓï¾ä

µÚËĽ׶ΣºSpark Streaming ->½øÐÐÁ÷ʽ¼ÆË㣺±ÈÈ磺×ÔÀ´Ë®³§

(4)Apache Storm ÀàËÆ£ºSpark Streaming ->½øÐÐÁ÷ʽ¼ÆËã

NoSQL£ºRedis»ùÓÚÄÚ´æµÄÊý¾Ý¿â

HDFS

·Ö²¼Ê½Îļþϵͳ ½â¾öÒÔÏÂÎÊÌ⣺

Ó²Å̲»¹»´ó£º¶à¼¸¿éÓ²ÅÌ£¬ÀíÂÛÉÏ¿ÉÒÔÎÞÏÞ´ó

Êý¾Ý²»¹»°²È«£ºÈßÓà¶È£¬hdfsĬÈÏÈßÓàΪ3 £¬ÓÃˮƽ¸´ÖÆÌá¸ßЧÂÊ£¬´«Êä°´ÕÕÊý¾Ý¿âΪµ¥Î»£ºHadoop1.x 64M£¬Hadoop2.x 128M

¹ÜÀíÔ±£ºNameNode Ó²ÅÌ£ºDataNode

![image.png]() 

MapReduce

»ù´¡±à³ÌÄ£ÐÍ£º°ÑÒ»¸ö´óÈÎÎñ²ð·Ö³ÉСÈÎÎñ£¬ÔÙ½øÐлã×Ü

MRÈÎÎñ£ºJob = Map + Reduce

MapµÄÊä³öÊÇReduceµÄÊäÈë¡¢MRµÄÊäÈëºÍÊä³ö¶¼ÊÇÔÚHDFS

MapReduceÊý¾ÝÁ÷³Ì·ÖÎö£º

MapµÄÊä³öÊÇReduceµÄÊäÈ룬ReduceµÄÊäÈëÊÇMapµÄ¼¯ºÏ

´óÊý¾ÝÈëÃÅÖ®Hadoop»ù´¡Ñ§Ï°

HBase

ʲôÊÇBigTable?: °ÑËùÓеÄÊý¾Ý±£´æµ½Ò»ÕűíÖУ¬²ÉÓÃÈßÓà ---> ºÃ´¦£ºÌá¸ßЧÂÊ

ÒòΪÓÐÁËbigtableµÄ˼Ï룺NoSQL£ºHBaseÊý¾Ý¿â

HBase»ùÓÚHadoopµÄHDFSµÄ

ÃèÊöHBaseµÄ±í½á¹¹

ºËÐÄ˼ÏëÊÇ£ºÀûÓÿռ任ЧÂÊ

´óÊý¾ÝÈëÃÅÖ®Hadoop»ù´¡Ñ§Ï°

Hadoop»·¾³´î½¨

»·¾³×¼±¸

Linux»·¾³¡¢JDK¡¢

°²×°

1¡¢°²×°jdk¡¢²¢ÅäÖû·¾³±äÁ¿

vim /etc/profile Ä©Î²Ìí¼Ó 

![image.png]() 

2¡¢½âѹhadoop-3.0.0.tar.gz¡¢²¢ÅäÖû·¾³±äÁ¿

tar -zxvf hadoop-3.0.0.tar.gz -C /usr/local

mv hadoop-3.0.0/ hadoop 

´óÊý¾ÝÈëÃÅÖ®Hadoop»ù´¡Ñ§Ï°

´óÊý¾ÝÈëÃÅÖ®Hadoop»ù´¡Ñ§Ï°

vim /etc/profile ĩβÌí¼Ó

´óÊý¾ÝÈëÃÅÖ®Hadoop»ù´¡Ñ§Ï°

ÅäÖÃ

HadoopÓÐÈýÖÖ°²×°Ä£Ê½£º

±¾µØÄ£Ê½ £º

1̨Ö÷»ú

²»¾ß±¸HDFS£¬Ö»ÄܲâÊÔMapReduce³ÌÐò

α·Ö²¼Ä£Ê½£º

1̨Ö÷»ú

¾ß±¸HadoopµÄËùÓй¦ÄÜ£¬ÔÚµ¥»úÉÏÄ£ÄâÒ»¸ö·Ö²¼Ê½µÄ»·¾³

(1)HDFS£ºÖ÷£ºNameNode£¬Êý¾Ý½Úµã£ºDataNode

(2)Yarn£ºÈÝÆ÷£¬ÔËÐÐMapReduce³ÌÐò

Ö÷½Úµã£ºResourceManager

´Ó½Úµã£ºNodeManager

È«·Ö²¼Ä£Ê½£º

ÖÁÉÙ3̨

ÎÒÃÇÒÔα·Ö²¼Ä£Ê½ÎªÀýÅäÖãº

ÐÞ¸Ähdfs-site.xml£ºÈßÓà¶È1¡¢È¨ÏÞ¼ì²éfalse

<!--ÅäÖÃÈßÓà¶ÈΪ1--> 

<property> 

    <name>dfs.replication</name

    <value>1</value> 

</property> 

 

<!--ÅäÖÃȨÏÞ¼ì²éΪfalse--> 

<property> 

    <name>dfs.permissions</name

    <value>false</value> 

</property> 

ÐÞ¸Äcore-site.xml

<!--ÅäÖÃHDFSµÄNameNode--> 

<property> 

    <name>fs.defaultFS</name

    <value>hdfs://192.168.56.102:9000</value> 

</property> 

 

<!--ÅäÖÃDataNode±£´æÊý¾ÝµÄλÖÃ--> 

<property> 

    <name>hadoop.tmp.dir</name

    <value>/usr/local/hadoop/tmp</value> 

</property> 

ÐÞ¸Ämapred-site.xml

<!--ÅäÖÃMRÔËÐеĿò¼Ü--> 

<property> 

    <name>mapreduce.framework.name</name

    <value>yar</value> 

</property> 

<property> 

    <name>yarn.app.mapreduce.am.env</name

    <value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value> 

</property> 

<property> 

    <name>mapreduce.map.env</name

    <value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value> 

</property> 

<property> 

    <name>mapreduce.reduce.env</name

    <value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value> 

</property> 

<property> 

    <name>mapreduce.application.classpath</name

    <value> 

            /usr/local/hadoop/etc/hadoop, 

            /usr/local/hadoop/share/hadoop/common/*, 

            /usr/local/hadoop/share/hadoop/common/lib/*, 

            /usr/local/hadoop/share/hadoop/hdfs/*, 

            /usr/local/hadoop/share/hadoop/hdfs/lib/*, 

            /usr/local/hadoop/share/hadoop/mapreduce/*, 

            /usr/local/hadoop/share/hadoop/mapreduce/lib/*, 

            /usr/local/hadoop/share/hadoop/yarn/*, 

            /usr/local/hadoop/share/hadoop/yarn/lib/*, 

    </value> 

</property> 

ÐÞ¸Äyarn-site.xml

<!--ÅäÖÃResourceManagerµØÖ·--> 

<property> 

    <name>yarn.resourcemanager.hostname</name

    <value>192.168.56.102</value> 

</property> 

 

<!--ÅäÖÃNodeManagerÖ´ÐÐÈÎÎñµÄ·½Ê½--> 

<property> 

    <name>yarn.nodemanager.aux-service</name

    <value>mapreduce_shuffle</value> 

</property> 

¸ñʽ»¯NameNode

hdfs namenode -format 

¿´µ½common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted±íʾ¸ñʽ»¯³É¹¦

Æô¶¯

start-all.sh 

(*)HDFS£º´æ´¢Êý¾Ý

(*)YARN£º

·ÃÎÊ

£¨*£©ÃüÁîÐР

£¨*£©Java Api 

£¨*£©WEB Console 

HDFS: :50070

Yarn: :8088

ÔĶÁÑÓÕ¹

1
3