Mais conteúdo relacionado Semelhante a ソーシャルアプリでの Amazon Elastic MapReduce 活用事例 (8) ソーシャルアプリでの Amazon Elastic MapReduce 活用事例2. •
•
• Amazon Elastic MapReduce
•
Sunday, September 26, 2010
3. • Twitter: @buhii
•
• gumi @ http://www.kansei.tsukuba.ac.jp/~uchiyamalab/beacon
•
• beacon
• gumi @ynil
Sunday, September 26, 2010
5. gumi
• mixi, , GREE
• python Django
• Amazon Web Services (EC2 + RDS)
•
• DB
Sunday, September 26, 2010
6. • PV , UU
• DAU Daily Active Users
•
•
• ÷ DAU
• ARPU Average Revenue Per User
•
Sunday, September 26, 2010
14. MapReduce
Mapper Key, Value
Mapper key
Sort / Shuffle Reducer
Reducer key, value
Mapper, Reducer
Sunday, September 26, 2010
15. Amazon Elastic MapReduce
• Hadoop
• Hadoop Streaming Mapper
Reducer Ruby, Perl, Python, PHP, R, Bash, C++
• EC2 job
•
Sunday, September 26, 2010
17. • Mapper
• Apache Log
• ID key
value
Reducer
Sunday, September 26, 2010
18. • Reducer
• sort/shuffle ID
Reducer
• ID
Sunday, September 26, 2010
19. Reducer
31758623 2010-08-20
42346572 2010-09-05,2010-09-06
31977736 2010-08-11,2010-08-12,2010-08-13,2010-08-14
14007991 2010-08-16
35995849 2010-08-12,2010-08-13,2010-08-14
34246688 2010-08-21,2010-08-22,2010-08-23,2010-08-27
...
PC
Sunday, September 26, 2010
20. Amazon Elastic Mapreduce
• AWS
• S3 Mapper, Reducer
→ s3cmd, S3Fox Organizer, Cyberduck
• Job OK
•
Sunday, September 26, 2010
24. {Input, Output} Location, Mapper, Reducer
S3
gzip
Hadoop Extra Args
-jobconf stream.recordreader.compression=gzip
input Location Extra Args
-input s3n://(bucket )/( )/access_log.*
Sunday, September 26, 2010
30. • Hadoop
• MapReduce
•
•
Sunday, September 26, 2010
31. Hadoop
•
• S3 gzip
•
• hadoop EC2
•
( 20 ...)
Sunday, September 26, 2010
32. @ynil
MapReduce
http://nlpyutori.g.hatena.ne.jp/yaruki_nil/20100911/1284089305
Sunday, September 26, 2010
36. MapReduce
MapReduce
Google
Map
Reduce Map
Reduce
MapReduce C++ Java Python
Wikipedia “MapReduce”
http://ja.wikipedia.org/wiki/MapReduce
Sunday, September 26, 2010
37. cron
•
PV, UU NFS CSV
• DB
→ DB
• PV, UU
Sunday, September 26, 2010