O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Real Time Processing Using Twitter Heron by Karthik Ramasamy

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio

Vídeos do YouTube não são mais aceitos pelo SlideShare

Visualizar original no YouTube

Low Latency Streaming Using Heron
Karthik Ramasamy
@karthikz
Co-founder of Streamlio
2
Real-time is key
Information Age
Ká

Confira estes a seguir

1 de 31 Anúncio

Real Time Processing Using Twitter Heron by Karthik Ramasamy

Abstract:- Today's enterprises are not only producing data in high volume but also at high velocity. With velocity comes the need to process the data in real time. To meet the real time needs, we developed and deployed Heron, the next generation streaming engine at Twitter. Heron processes billions and billions of events per day at Twitter and has been in production for nearly 3 years. Heron provides unparalleled performance at large scale and has been successfully meeting Twitter's strict performance requirements for various streaming and iOT applications. Heron is a open source project with several major contributors from various institutions. As the project, we identified and implemented several optimizations that improved throughput by additional 5x and further reduce latency by 50-60%. In this talk, we will describe Heron in detail, how the detailed profiling indicated the performance bottleneck areas such as multiple serializations/deserialization and immutable data structures. After mitigating these costs, we were able to show much higher throughput and latencies as low as 12ms.

Abstract:- Today's enterprises are not only producing data in high volume but also at high velocity. With velocity comes the need to process the data in real time. To meet the real time needs, we developed and deployed Heron, the next generation streaming engine at Twitter. Heron processes billions and billions of events per day at Twitter and has been in production for nearly 3 years. Heron provides unparalleled performance at large scale and has been successfully meeting Twitter's strict performance requirements for various streaming and iOT applications. Heron is a open source project with several major contributors from various institutions. As the project, we identified and implemented several optimizations that improved throughput by additional 5x and further reduce latency by 50-60%. In this talk, we will describe Heron in detail, how the detailed profiling indicated the performance bottleneck areas such as multiple serializations/deserialization and immutable data structures. After mitigating these costs, we were able to show much higher throughput and latencies as low as 12ms.

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Real Time Processing Using Twitter Heron by Karthik Ramasamy (20)

Anúncio

Mais de Data Con LA (20)

Mais recentes (20)

Anúncio

Real Time Processing Using Twitter Heron by Karthik Ramasamy

  1. 1. Low Latency Streaming Using Heron Karthik Ramasamy @karthikz Co-founder of Streamlio
  2. 2. 2 Real-time is key Information Age Ká
  3. 3. 3 Real Time Connected World Internet of Things 30 B connected devices by 2020 Health Care 153 Exabytes (2013) -> 2314 Exabytes (2020) Machine Data 40% of digital universe by 2020 Connected Vehicles Data transferred per vehicle per month 4 MB -> 5 GB Digital Assistants (Predictive Analytics) $2B (2012) -> $6.5B (2019) [1] Siri/Cortana/Google Now Augmented/Virtual Reality $150B by 2020 [2] Oculus/HoloLens/Magic Leap Ñ + > [1] http://www.siemens.com/innovation/en/home/pictures-of-the-future/digitalization-and-software/digital-assistants-trends.html [2] http://techcrunch.com/2015/04/06/augmented-and-virtual-reality-to-hit-150-billion-by-2020/#.7q0heh:oABw
  4. 4. 4 Value of Data Value&of&Data&to&Decision/Making& Time& Preven8ve/& Predic8ve& Ac8onable& Reac8ve& Historical& Real%& Time& Seconds& Minutes& Hours& Days& Tradi8onal&“Batch”&&&&&&&&&&&&&&& Business&&Intelligence& Informa9on&Half%Life& In&Decision%Making& Months& Time/cri8cal& Decisions& [1] Courtesy Michael Franklin, BIRTE, 2015.
  5. 5. 5 Introducing Heron ! Scaling ! Debugging ! Consistent performance ! Yet another system to manage ! Consistent performance at scale ! Easy to debug and tune ! Fast/Efficient General purpose streaming engine ! Storm API compatibile ! Latency/Thruput configurability ! Library not a service Issues with Apache Storm Heron Design Goals
  6. 6. 6 Heron in Production @ Twitter Completely replaced Storm 3 years ago 3x reduction in cores and memory Significantly reduced operational overhead 10x reduction in production incidents
  7. 7. 7 Heron Use Cases REALTIME ETL REAL TIME BI SPAM DETECTION REAL TIME TRENDS REALTIME ML REAL TIME OPS
  8. 8. 8 Open Souring https://github.com/twitter/heron http://heron.io Apache 2.0 License Contributions from Microsoft, Machine Zone, Mesosphere, Google, Wanda Group, WeChat, Fitbit and growing OPEN SOURCED MAY 2016
  9. 9. 9 Heron Core Concepts Topology Directed acyclic graph vertices = computation, and edges = streams of data tuples Spouts Sources of data tuples for the topology Examples - Kafka/Kestrel/MySQL/Postgres Bolts Process incoming tuples, and emit outgoing tuples Examples - filtering/aggregation/join/any function , %
  10. 10. 10 Sample Heron Topology % % % % % Spout 1 Spout 2 Bolt 1 Bolt 2 Bolt 3 Bolt 4 Bolt 5
  11. 11. 11 Topology Architecture Topology Master ZK Cluster Stream Manager I1 I2 I3 I4 Stream Manager I1 I2 I3 I4 Logical Plan, Physical Plan and Execution State Sync Physical Plan CONTAINER CONTAINER Metrics Manager Metrics Manager
  12. 12. 12 Stream Manager - Design Goals Core logic in one centralized place Super Efficient Pluggable Transport (tcp sockets, unix sockets, shared memory) Interlanguage Data Format (Protobufs, Cap N’ Proto, etc) Protocol (HTTP, gRPC, custom, etc) Oculus/HoloLens/Magic Leap Ñ + > Multilanguage Instances (C++, Java, Python),
  13. 13. 13 Stream Manager - Current Implementation Implements at most once and at least once Written in C++ for efficiency Custom protocol (very similar to gRPC) Transport using TCP Sockets Protobuf data format Ñ +
  14. 14. 14 Stream Manager - Shortcomings 01 02 03 Transport Shortcomings TCP Overhead Multiple memory copies Protobuf Shortcomings Serde is very expensive Full deserialization necessary to access any field Creation/Deletion is very expensive Core Logic Implementation Followed immutable pattern Easy to reason but inefficient / . -
  15. 15. 15 Stream Manager - Performance Analysis Too slow Too much overhead Changes what we are trying to observe Very fast Doesn’t do code instrumentation cpu-profiling/memory-profiling in one tool Ñ Valgrind Google Perftools
  16. 16. 16 Stream Manager - Performance Analysis 17% in new/delete 15% immutable pattern 15% eager deserialization 12% protobuf size collection
  17. 17. 17 Stream Manager - Optimization 1 ! new/delete overhead ! Problem:- Create/Delete a new protobuf object every time we read/wrote something. ! Protobuf sacrifices speed for safety ! Solution ! Create protobuf pools at startup ! We do a “new” only when the pool is exhausted ! The pool is bounded in size to avoid running out of memory
  18. 18. 18 Stream Manager - Optimization 2 ! Immutable pattern ! Problem:- In general case, one tuple can fan-out to multiple downstream instances ! For each downstream instance, we made a new copy ! Solution ! Do early serialization to create an immutable byte array ! Just copy the raw bytes
  19. 19. 19 Stream Manager - Optimization 3 ! Eager Deserialization ! Problem:- Protobuf deserializes the entire message even if we access just the ‘header’ ! Solution ! Change the protobuf message to have raw bytes to avoid expensive deserialization ! Lazy deserialization is done manually only when needed
  20. 20. 20 Stream Manager - Optimization 4 ! Calculation of Protobuf ByteSize ! Problem:- Bytesize computation is expensive and every time the computation is from the scratch ! Solution ! Used CachedByteSize when possible
  21. 21. 21 Benchmark Settings Components Expt 1 Expt 2 Expt 3 Spout 25 100 200 Bolt 25 100 200 # Heron Containers 25 100 200 Dual Intel Xeon E5645@2.4GHz, 72GB RAM, 500GB Disk 175K Random words generated Word Count Topology
  22. 22. 22 Benchmark - At most once throughput 5 - 6x
  23. 23. 23 Benchmark - At least once throughput 4 - 5x
  24. 24. 24 Benchmark - At least once latency 2 - 4x
  25. 25. 25 Real-Time is Messy, Unpredictable and Hard Aggregation Systems Messaging Systems Result Engine HDFS Queryable Engines
  26. 26. 26 Real Time - End to End Storm API DSL SQL Application Builder Ingestion API Query API
  27. 27. 27 Curious to Learn More? Twitter Heron: Stream Processing at Scale Sanjeev Kulkarni, Nikunj Bhagat, Maosong Fu, Vikas Kedigehalli, Christopher Kellogg, Sailesh Mittal, Jignesh M. Patel*,1 , Karthik Ramasamy, Siddarth Taneja @sanjeevrk, @challenger_nik, @Louis_Fumaosong, @vikkyrk, @cckellogg, @saileshmittal, @pateljm, @karthikz, @staneja Twitter, Inc., *University of Wisconsin – Madison ABSTRACT Storm has long served as the main platform for real-time analytics at Twitter. However, as the scale of data being processed in real- time at Twitter has increased, along with an increase in the diversity and the number of use cases, many limitations of Storm have become apparent. We need a system that scales better, has better debug-ability, has better performance, and is easier to manage – all while working in a shared cluster infrastructure. We considered various alternatives to meet these needs, and in the end concluded that we needed to build a new real-time stream data processing system. This paper presents the design and implementation of this new system, called Heron. Heron is now the de facto stream data processing engine inside Twitter, and in this paper we also share our experiences from running Heron in production. In this paper, we also provide empirical evidence demonstrating the efficiency and scalability of Heron. ACM Classification H.2.4 [Information Systems]: Database Management—systems Keywords Stream data processing systems; real-time data processing. 1. INTRODUCTION Twitter, like many other organizations, relies heavily on real-time system process, which makes debugging very challenging. Thus, we needed a cleaner mapping from the logical units of computation to each physical process. The importance of such clean mapping for debug-ability is really crucial when responding to pager alerts for a failing topology, especially if it is a topology that is critical to the underlying business model. In addition, Storm needs dedicated cluster resources, which requires special hardware allocation to run Storm topologies. This approach leads to inefficiencies in using precious cluster resources, and also limits the ability to scale on demand. We needed the ability to work in a more flexible way with popular cluster scheduling software that allows sharing the cluster resources across different types of data processing systems (and not just a stream processing system). Internally at Twitter, this meant working with Aurora [1], as that is the dominant cluster management system in use. With Storm, provisioning a new production topology requires manual isolation of machines, and conversely, when a topology is no longer needed, the machines allocated to serve that topology now have to be decommissioned. Managing machine provisioning in this way is cumbersome. Furthermore, we also wanted to be far more efficient than the Storm system in production, simply because at Twitter’s scale, any improvement in performance translates into significant reduction in infrastructure costs and also significant improvements in the productivity of our end users. Storm @Twitter Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy, Jignesh M. Patel*, Sanjeev Kulkarni, Jason Jackson, Krishna Gade, Maosong Fu, Jake Donham, Nikunj Bhagat, Sailesh Mittal, Dmitriy Ryaboy @ankitoshniwal, @staneja, @amits, @karthikz, @pateljm, @sanjeevrk, @jason_j, @krishnagade, @Louis_Fumaosong, @jakedonham, @challenger_nik, @saileshmittal, @squarecog Twitter, Inc., *University of Wisconsin – Madison Streaming@Twitter Maosong Fu, Sailesh Mittal, Vikas Kedigehalli, Karthik Ramasamy, Michael Barry, Andrew Jorgensen, Christopher Kellogg, Neng Lu, Bill Graham, Jingwei Wu Twitter, Inc. Abstract Twitter generates tens of billions of events per hour when users interact with it. Analyzing these events to surface relevant content and to derive insights in real time is a challenge. To address this, we developed Heron, a new real time distributed streaming engine. In this paper, we first describe the design goals of Heron and show how the Heron architecture achieves task isolation and resource reservation to ease debugging, troubleshooting, and seamless use of shared cluster infrastructure with other critical Twitter services. We subsequently explore how a topology self adjusts using back pressure so that the pace of the topology goes as its slowest component. Finally, we outline how Heron implements at most once and at least once semantics and we describe a few operational stories based on running Heron in production. 1 Introduction Stream processing platforms enable enterprises to extract business value from data in motion similar to batch processing platforms that facilitated the same with data at rest [42]. The goal of stream processing is to enable real time or near real time decision making by providing capabilities to inspect, correlate and analyze data as
  28. 28. 28 Curious to Learn More?
  29. 29. 29 WHAT WHY WHERE WHEN WHO HOW Any Question ???
  30. 30. 30 @karthikz Get in Touch

×