Storm: Distributed and fault tolerant realtime computation
•
1 gostou•1,283 visualizações
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (14)
Semelhante a Storm: Distributed and fault tolerant realtime computation
Semelhante a Storm: Distributed and fault tolerant realtime computation (20)
Último
Último (20)
Storm: Distributed and fault tolerant realtime computation
- 1. Storm: Distributed and fault-tolerant realtime computation Ferran Galí i Reniu @ferrangali 19/06/2014
- 2. Ferran Galí i Reniu ● UPC - FIB ● Trovit ○ Hadoop ○ Lucene/Solr ○ Storm
- 3. Big Data ● Too much data ○ Store ○ Compute ○ Analyse ● Distributed systems ○ Provide horizontal scalability
- 4. ● Hadoop Distributed Systems HDFS HDFS HDFS File
- 6. Distributed Systems ● Hadoop ○ Huge files ○ Useful for batch ○ High latency ○ No real time
- 8. Storm “Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use!” http://storm.incubator.apache.org/
- 9. Storm ● Who’s using it?
- 10. ● Tuple ○ Ordered list of elements ○ Any type Storm String Integer Serialized Object ...
- 11. Storm ● Stream ○ Unbounded sequence of tuples Tuple Tuple Tuple Tuple Tuple Tuple Tuple
- 12. Storm ● Spout ○ Source of streams ○ From data sources: Queues, API... Tuple Tuple Tuple Tuple Tuple
- 13. Storm ● Bolt ○ Consumes streams ○ Does some processing (transform, join,...) ○ Emits streams Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple Tuple
- 14. Storm ● Topology ○ Graph of spouts & bolts ○ Runs forever
- 17. parallelism hint = 4 parallelism hint = 1 parallelism hint = 2 parallelism hint = 2 parallelism hint = 3 parallelism hint = 4 Supervisor Slot Slot Slot Slot Supervisor Slot Slot Slot Slot Worker processes = 8
- 18. parallelism hint = 4 parallelism hint = 1 parallelism hint = 2 parallelism hint = 2 parallelism hint = 3 parallelism hint = 4 Worker processes = 8 combined parallelism = 4 + 1 + 2 + 2 + 3 + 4 = 16 Tasks per worker = 16 / 8 = 2 Supervisor Supervisor
- 19. Example: Word Count line line line word word word File FileSpout SplitterBolt CounterBolt parallelism hint = 2 parallelism hint = 3 parallelism hint = 2
- 20. SplitterBoltFileSpout Example: Word Count CounterBolt Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use!
- 21. SplitterBoltFileSpout Example: Word Count CounterBolt Storm is a distributed Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use!
- 22. SplitterBoltFileSpout Example: Word Count CounterBolt Storm is a distributed Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use! realtime computation system. Storm provides a
- 23. SplitterBoltFileSpout Example: Word Count CounterBolt Storm is a distributed Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use! realtime computation system. Storm provides a shuffle grouping
- 24. SplitterBoltFileSpout Example: Word Count CounterBolt Storm is a distributed Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use! realtime computation system. Storm provides a Storm a is distributed realtime computation system provides Storm a shuffle grouping
- 25. SplitterBoltFileSpout Example: Word Count CounterBolt Storm is a distributed Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use! realtime computation system. Storm provides a Storm a is distributed realtime computation system provides Storm a Storm a is distributed realtime computation system provides Storm a x1 x1 x1 x1 x1 x1 x1 x1 x1 x1 shuffle grouping
- 26. SplitterBoltFileSpout Example: Word Count CounterBolt Storm is a distributed Storm is a distributed realtime computation system. Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use! realtime computation system. Storm provides a shuffle grouping a is Storm distributed provides a Storm is distributed realtime computation system a x2 x1 x1 x1 x2 x1 x1 x1 realtime computation provides fields grouping system Storm
- 27. Groupings ● Shuffle grouping ● Fields grouping ● All grouping ● Global grouping ● Direct grouping ● Local or shuffle grouping
- 29. ● Worker dies ○ Supervisor will restart it ● Worker dies too many times ○ Nimbus will reassign it to another node ● Node dies ○ Nimbus will reassign task to another node ● Nimbus is not a SPOF ● Nimbus & Supervisors are fail-fast Fault-tolerance
- 30. Guaranteeing message processing ● Through API ○ ack ○ fail ● Manual tuple replay ○ e.g: Spout emits again message with specific id
- 31. Guaranteeing message processing ● When is a message “fully processed”? ● Solutions ○ Transactional Topologies ○ Trident framework Storm is a distributed Storm is distributed a Ok Fail Ok Ok
- 32. Yet another example tweet tweet tweet word word word TwitterSpout SplitterBolt CounterBolt CommitBolt signal signal signal DB shuffle grouping fields grouping all grouping https://github.com/ferrangali/betabeers-storm
- 33. Batch + Real time ● Lambda architecture Serving Batch layer ● High latency ● Reprocesses all data New data
- 34. Batch + Real time ● Lambda architecture Speed layer Serving Batch layer ● Low latency ● Fast & incremental algorithms ● Eventually overridden by batch layer ● High latency ● Reprocesses all data New data
- 35. Storm ● Who’s using it?
- 36. Trovit ● 40 countries ● 5 verticals ● Hundreds of millions of ads
- 37. Trovit ● Batch layer: ○ MapReduce pipeline over HDFS HDFS Filter Enrich Dedup Index kafka xml
- 38. Trovit ● Speed layer ○ Storm topology ad ad ad ad ad ad rich ad rich ad rich ad Feeds Spout Kafka Spout Processor Bolt Indexer Bolt Group by index Commit in batch every 5 minutes kafka xml
- 39. Trovit HDFS Filter Enrich Dedup Index ad ad ad ad ad ad richad richad richad HBaseZookeeper kafka xml
- 40. Questions? Ferran Galí i Reniu @ferrangali 19/06/2014