Enterprise Grade Streaming under 2ms on Hadoop
•Transferir como PPTX, PDF•
0 gostou•571 visualizações
Enterprise Grade Streaming under 2ms on Hadoop
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Enterprise Grade Streaming under 2ms on Hadoop
Semelhante a Enterprise Grade Streaming under 2ms on Hadoop (20)
Mais de DataWorks Summit/Hadoop Summit
Mais de DataWorks Summit/Hadoop Summit (20)
Último
Último (20)
Enterprise Grade Streaming under 2ms on Hadoop
- 1. Enterprise Grade Streaming Under 2ms On Hadoop @vijaysbhat
- 2. 2
- 3. 3 VS.
- 4. 4
- 5. 5
- 6. 6
- 7. 7 X (predictor) Spend amount, geo Y (response) Simple Velocity Advanced
- 8. 8
- 9. 9
- 10. 10
- 11. 11 Hard Metrics Goal Latency < 40ms Ideally < 16ms Throughput Goal of 2000 events / second Durability No loss, every message gets exactly one response Availability 99.5% uptime (downtime of 1.83 days / year); Ideally 99.999% uptime (downtime of 5.26 minutes / year) Scalability Can add resources, still meet latency requirements Integration Transparently connected to existing systems – Hardware, Messaging, HDFS Soft Metrics Goal Open Source All components licensed as open source Extensibility Rules can be updated, model is regularly refreshed
- 12. 12
- 13. 13 Onyx
- 15. 15
- 16. 16
- 17. 17
- 18. 18
- 19. 19
- 20. 20
- 21. 21
- 22. 22
- 23. 23
- 25. 25
- 26. 26 • Avg. 0.25ms, @70k records/sec, w/ 600GB RAM Thread Local on ~54M events Percentiles (in ms) Throughput Count Avg (ms) 90% 95% 99% 99.9% 4 9’s 5 9’s 6 9’s 70k/sec 54,126,1 22 0.19 1 1 1 2 2 5 6 Performance
- 27. 27 Durability • Two physically independent pipelines on the same cluster processing identical data • For the same tuple, we find the best-case time between two pipelines – 39 records out of 5.2M exceeded 16ms – 173 out of 5.2M exceeded 16ms in one pipeline but succeeded in the other • 99.99925% success rate – “Five Nines” • Average Latency of 0.0981ms
Notas do Editor
- Talk about next generation real time decisioning system
- Batch systems Scaling out challenging but no strict time requirements Cannot operate on one piece of data at a time Large latency
- Mainframes High performance systems Highly specialized Very expensive, but extremely durable Open source Customize code Grow project organically Not going to get same level of performance
- Availability is crucial Measure of downtime what fraction will be unavailable Measured in nines – 5 nines is 5 minutes
- Free More flexible Can build yourself Future proof design
- What we do today Credit card swiped, take decision on it based on model. Bunch of inputs produces decision Executed on mainframe
- Simple features Aggregate features Need notion of state Need in memory DB due to disk latency of 100ms Advanced features Joining multiple sources
- Current situation - Proprietary solution, slow refresh, costs a lot of money - Data is stale, models are stale, quality of decisions is low, throwing money away Swamp, don’t own anything. Mordor is dark and terrifying. Just throwing money at the swamp.
- - How do we get to a better place?
- Prove whatever we are doing is valid for the use case – have to convince business stakeholders that we have the right solution. - Need rigor
- Open source Keep options open not stuck with proprietary solution Take our learnings give back to community Future proof what we do
- Infosphere – built for NSA, handles video and audio traffic on a global scale. Hazelcast – gridgain ignite. Onyx – streaming platform built in Clojure Feedzai – proprietary Storm Cassandra stack
- Performance – durability availability latency Enterprise ready – measure of confidence Roadmap – private conversations with folks behind platforms Community – how vibrant is the community, how is the adoption
- We chose Apex, and I’ll go ahead and explain why
- Not true streaming, microbatching 100s of ms latency unavoidable even with small window size Great for offline ETL and we use it for computing some of our slow moving features
- Spark streaming is missing – nonstarter due to microbatch, lack of dynamic dag reconfiguration
- Failure detection through ACK which cannot be configured lower than 1 second Replay from source, means for durability, need to have separate applications. Resource sharing not great on Storm which is why Twitter built Heron Fast Easy to use Mature *********** Failures are not independent, nimbus, no dynamic topologies, 1 sec ack, resource usage Community stagnating, only Horton Roadmap still to bring it to enterprise (Integration with YARN, elastic topologies, high availability)
- How does Flink handle failures? Inject barriers / markers and checkpoint against them to disk Replay from last global checkpoint – still need to load from disk Reset whole pipeline on failure – can’t have truly independent pipelines Baidu running 1000 node cluster Easy to Use Fast Support for SQL-like queries ----- Meeting Notes (10/27/15 10:14) ----- Reset to upstream data source, Shared JVM, No dynamic topologies Young community Roadmap – fine grained fault tolerance, in-memory store integration, off-heap memory, full SQL
- Brings us to Apex
- Veterans from Yahoo! Finance and Hadoop Built for Enterprise stability and durability before performance ***************** Phu Hoang - CEO and Co-Founder head of engineering at Yahoo Amol Kekre - Led Yahoo! Finance and Led YARN Chetan - Lead architect from Yahoo Finance Thomas Weise - Hadoop Veteran from Yahoo
- Dynamic modifying topologies on a live system without affecting performance Also supports dynamic partitioning when data gets skewed
- Fine grained control of locality for streams thread, node, container, rack Can define both affinity and anti-affinity
- Independent pipelines deployed in the same application. Buffer server offers fine grained state persistence so whole topology doesn’t need to be reset
- Different pieces of data (partitions) going through same logical operator by mapping it to physical instances of operator Independence of partitions Auto-scaling (throughput and latency)
- 10 node cluster
- - If either one of them responds within 16 ms we have won