Hadoop ecosystem

Moscow, November 16th, 2011 The Hadoop Ecosystem Kai Voigt , Cloudera Inc.

Cloudera ©2011 Cloudera, Inc. All Rights Reserved. Hadoop Linux Licence Apache GPL and others Distribution Vendor Cloudera Red Hat Free Distribution Cloudera's Distribution Including Hadoop (CDH) Fedora Core Commercial Distribution Cloudera Enterprise Red Hat Enterprise Linux (RHEL)

HDFS-FUSE Examples ©2011 Cloudera, Inc. All Rights Reserved. $ mount ... fuse on /mnt/hdfs type fuse (rw,nosuid,nodev,user_id=0,group_id=0,default_permissions,allow_other) $ cp /boot/vmlinuz-* /mnt/hdfs/user/cloudera/ $ hadoop fs -ls vmlinuz-*-rw-r--r-- 3 cloudera supergroup 2107004 2011-11-08 16:14 /user/cloudera/vmlinuz-2.6.18-274.7.1.el5

Sqoop Examples ©2011 Cloudera, Inc. All Rights Reserved. $ sqoop import --connect jdbc:mysql://localhost/world --username root --table City ... $ hadoop fs -cat City/part-m-00000 1,Kabul,AFG,Kabol,17800002,Qandahar,AFG,Qandahar,2375003,Herat,AFG,Herat,1868004,Mazar-e-Sharif,AFG,Balkh,1278005,Amsterdam,NLD,Noord-Holland,731200 ...

Hive Examples ©2011 Cloudera, Inc. All Rights Reserved. CREATE TABLE newmovie (id INT, name STRING, year INT, numratings INT, avgrating FLOAT);INSERT OVERWRITE TABLE newmovieSELECT id, name, year, COUNT(1), AVG(rating)FROM movie JOIN movieratingON movie.id = movierating.movieidGROUP BY id, name, year;

Pig Examples ©2011 Cloudera, Inc. All Rights Reserved. movierating = LOAD 'movierating' AS (userid, movieid, rating:INT);groupmr = GROUP movierating BY movieid;ratings = FOREACH groupmr GENERATE group AS movieid, COUNT(movierating.rating) AS numratings, AVG(movierating.rating) AS avgrating;movie = LOAD 'movie' AS (id, name, year);mr = JOIN movie BY id, ratings BY movieid;result = FOREACH mr GENERATE id, name, year, numratings, avgrating;STORE result INTO 'ratedmovie';

HBase Data Model ©2011 Cloudera, Inc. All Rights Reserved. Key RowID Columname Timestamp Value com.apple.www Size yesterday 1234 com.apple.www Content yesterday <html>... com.cloudera.www Size yesterday 2345 com.cloudera.www Content yesterday <html>... com.cloudera.www Size today 3456 com.cloudera.www Content today <html>... com.facebook.www Size yesterday 4567 com.facebook.www Content yesterday <html>... com.yahoo.www Size today 5678 com.yahoo.www Content today <html>...

HBase Examples ©2011 Cloudera, Inc. All Rights Reserved. hbase> create 'mytable', 'mycf'hbase> listhbase> put 'mytable', 'row1', 'mycf:col1', 'val1'hbase> put 'mytable', 'row1', 'mycf:col2', 'val2'hbase> put 'mytable', 'row2', 'mycf:col1', 'val3'hbase> scan 'mytable'hbase> disable 'mytable'hbase> drop 'mytable'

Whirr Example ©2011 Cloudera, Inc. All Rights Reserved. $ cat hadoop.properties whirr.cluster-name=myhadoopcluster whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,7 hadoop-datanode+hadoop-tasktracker whirr.provider=aws-ec2 whirr.identity=${env:AWS_ACCESS_KEY_ID} whirr.credential=${env:AWS_SECRET_ACCESS_KEY} whirr.private-key-file=${sys:user.home}/.ssh/id_rsa whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub $ bin/whirr launch-cluster --config hadoop.properties $ . ~/.whirr/myhadoopcluster/hadoop-proxy.sh $ export HADOOP_CONF_DIR=~/.whirr/myhadoopcluster $ bin/whirr destroy-cluster --config hadoop.properties

Oozie Concept ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],©2011 Cloudera, Inc. All Rights Reserved. Job 1 Job 3 Job 2 Job 4 Job 5

Hadoop ecosystem

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Hadoop ecosystem

Semelhante a Hadoop ecosystem (20)

Mais de tfmailru

Mais de tfmailru (12)

Último

Último (20)

Hadoop ecosystem