Large scale data collection and processing pipeline Honu
•
26 likes•3,368 views
Hadoop Summit 2010 - Developers Track Honu - A Large Scale Streaming Data Collection and Processing Pipeline Jerome Boulon, Netflix
![Honu: A large scale data collection and processing pipeline ,[object Object],Netflix](https://image.slidesharecdn.com/7honuhadoopsummit2010-100630134425-phpapp02/85/honu-a-large-scale-streaming-data-collection-and-processing-pipelinehadoopsummit2010-1-320.jpg)
![[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Session Agenda](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Large scale data collection and processing pipeline Honu
Similar to Large scale data collection and processing pipeline Honu (20)
More from Yahoo Developer Network
More from Yahoo Developer Network (20)
Recently uploaded
Recently uploaded (20)
Large scale data collection and processing pipeline Honu
- 5. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 6. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 7. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 8. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 9. Architecture - Overview Application Collector Hive Data collection pipeline Data processing pipeline M/R
- 10. Current Netflix deployment EC2/EMR Applications S3 Hive MetaStore Hive & Hadoop EMR clusters Hive & Hadoop EMR (for ad-hoc query) Honu Collectors AMAZON EC2
- 14. Structured Log API – Using Annotation @ Resource (table=”MyTable") public class MyClass implements Annotatable { @ Column ("movieId") public String getMovieId() { […] } @Column("clickIndex") public int getClickIndex() { […] } @Column("requestInfo") public Map getRequestInfo() { […] } } DB=MyTable Movied=XXXX clieckIndex=3 requestInfo.returnCode=200 requestInfo.duration_ms=300 requestInfo.yyy=zzz log.info(myAnnotatableObj);
- 15. Structured Log API - Key/Value API KeyValueSerialization kv ; kv = new KeyValueSerialization(); […] kv.startMessage("MyTable"); kv.addKeyValue("movieid", ”XXX"); kv.addKeyValue("clickIndex", 3); kv.addKeyValue(”requestInfo", requestInfoMap); DB=MyTable Movied=XXXX clickIndex=3 requestInfo.returnCode=200 requestInfo.duration_ms=300 requestInfo.yyy=zzz log.info(kv.generateMessage());
- 17. Data Processing pipeline Application Collector Hive Data collection pipeline Data processing pipeline M/R
Editor's Notes
- This is the Title slide. Please use the name of the presentation that was used in the abstract submission.
- This is the agenda slide. There is only one of these in the deck.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.
- This is the final slide; generally for questions at the end of the talk. Please post your contact information here.