Resilient Predictive Data Pipelines (GOTO Chicago 2016)
•
2 gostaram•1,484 visualizações
Talk about Resilient Predictive Data Pipelines at the GOTO Chicago 2016 in the Always On track.
![About Me
2
Work [ed | s] @
Committer &
PPMC on
Report to
Co-Chair for](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (9)
Semelhante a Resilient Predictive Data Pipelines (GOTO Chicago 2016)
Semelhante a Resilient Predictive Data Pipelines (GOTO Chicago 2016) (20)
Mais de Sid Anand
Mais de Sid Anand (18)
Último
Último (20)
Resilient Predictive Data Pipelines (GOTO Chicago 2016)
- 1. Resilient Predictive Data Pipelines Sid Anand (@r39132) GOTO Chicago 2016 1
- 2. About Me 2 Work [ed | s] @ Committer & PPMC on Report to Co-Chair for
- 3. Motivation Why is a Data Pipeline talk in this Always Available Track? 3
- 4. Motivation 4 Always On work has traditionally focused on the availability of Serving Systems : • Synchronous or Semi-Synchronous • Often Transactional • Latency-sensitive
- 5. HA Goals of Serving Systems 5 Outages are Big News Items!
- 6. And sometimes your failures become your brand! 6
- 7. Motivation 7 Always On work has traditionally focused on the availability of Serving Systems : • Synchronous or Semi-Synchronous • Often Transactional • Latency-sensitive
- 8. Motivation 8 Arguably, the more valuable parts of online services are driven by Data Flow Systems (a.k.a. Data Pipelines): • Asynchronous • Throughput-sensitive
- 10. Serving + Data Pipelines 10 A business’s viability is based on its ability to • keep the site up (Always-On Serving Architectures) & • maintain engagement (views & clicks) with customers (Always- On Data Pipelines) This talk is about Always On Data Pipelines!
- 11. Serving + Data Pipelines 11 Serving Data Pipeline Data Integration Layer Web Servers Microservice Layer Data Layer (DB, Search, Caching, Graph DB, Object Store) FE Load Balancers DAGs + Scheduler + Distributed Computation Engine
- 13. Data Pipeline Challenges Problem 1 : The Blast Radius Problem 13
- 14. The Blast Radius Problem 14 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B
- 15. The Blast Radius Problem 15 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B • Data Pipeline Job 2 reads Data B & writes Data C
- 16. The Blast Radius Problem 16 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B • Data Pipeline Job 2 reads Data B & writes Data C • Data Pipeline Job 3 reads Data C & writes Data D to a Serving System DB
- 17. The Blast Radius Problem 17 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B • Data Pipeline Job 2 reads Data B & writes Data C • Data Pipeline Job 3 reads Data C & writes Data D to a Serving System DB • Serving System 4 reads Data D, where the bug is discovered!
- 18. The Blast Radius Problem 18 •The previous diagram only shows one path of a tree •The reality is much worse •For each data set produced, there are multiple consuming jobs and hence multiple bad downstream outputs
- 19. The Blast Radius Problem An acute pain point 19
- 20. The Blast Radius Problem 20 Detect Bug Job 1 Job 2 Job 3 Serving System
- 21. The Blast Radius Problem 21 Detect Bug Identify Cause
- 23. The Blast Radius Problem 23 Detect Bug Identify Cause
- 24. The Blast Radius Problem 24 Detect Bug Identify Cause Deploy a Fix
- 25. Rollout a Fix & Rerun all Downstream Jobs in the Affected Time Window 25
- 26. The Blast Radius Problem 26 Detect Bug Identify Cause Re_Run All Jobs over a Time Window
- 27. Take Aways? 27 • The cost in people, time, and morale for a Data Pipeline bug is high and they can occur frequently. • In most areas of software, testing is invaluable, less so in data pipelines • Data Pipeline bugs can be due to a logic problem or bad input data! • Best Option : Detect & Rollback/Fix Forward
- 28. The Blast Radius Solution 28 Detect Bug Identify Cause Re_Run 1 Job
- 30. Timeliness Job 1 Job 2 Job 3 Definition : job = workflow = DAG of tasks Job 3’s output is pushed to a serving system
- 31. Timeliness Run 1 : Monday Run 2 : Tuesday Run 3 : Wednesday Consider the Daily Run Schedule below:
- 32. 32 Timeliness Within Time SLA OUT Run 4 Run 5 R u n 6
- 33. Timeliness Why do jobs get slower? 33
- 34. 34 Timeliness
- 40. Take Aways? 40 • Data Science & Engineering work is a virtuous cycle of adding features (and the like) + tuning performance • Latency does matter (a bit)
- 41. Design Goals Desirable Qualities of a Resilient Data Pipeline 41
- 42. 42 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 43. 43 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • Data Integrity (no loss, etc…) • Expected data distributions • All output within time-bound SLAs • Fine-grained Monitoring & Alerting of Correctness & Timeliness SLAs • Quick Recoverability • Pay-as-you-go
- 44. Quickly Recoverable 44 • Bugs happen! • Bugs in Predictive Data Pipelines have a large blast radius • Optimize for MTTR
- 45. Implementation Using AWS to meet Design Goals 45
- 47. SQS - Overview 47 AWS’s low-latency, highly scalable, highly available message queue Infinitely Scalable Queue (though not FIFO) Low End-to-end latency (generally sub-second) Pull-based
- 48. visibility timer SQS - Typical Operation Flow 48 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 1: A consumer reads a message from SQS. This starts a visibility timer!
- 49. visibility timer SQS - Typical Operation Flow 49 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 2: Consumer persists message contents to DB
- 50. visibility timer SQS - Typical Operation Flow 50 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 3: Consumer ACKs message in SQS
- 51. visibility timer SQS - Time Out Example 51 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 1: A consumer reads a message from SQS
- 52. visibility timer SQS - Time Out Example 52 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 2: Consumer attempts persists message contents to DB
- 53. visibility time out SQS - Time Out Example 53 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 3: A Visibility Timeout occurs & the message becomes visible again.
- 54. visibility timer SQS - Time Out Example 54 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 m1 SQS Step 4: Another consumer reads and persists the same message
- 55. visibility timer SQS - Time Out Example 55 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 5: Consumer ACKs message in SQS
- 56. SQS - Dead Letter Queue 56 SQS - DLQ visibility timer Producer Producer Producer m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Redrive rule : 2x m1
- 58. SNS - Overview 58 Highly Scalable, Highly Available, Push-based Topic Service Whereas SQS is pull-based, SNS is push-based There is no message retention & there is a finite retry count No Reliable Message Delivery Whereas SQS ensures each message is seen by at least 1 consumer SNS ensures that each message is seen by every consumer Reliable Multi-Push Can we work around this limitation while getting Reliable Multi-push?
- 59. SNS + SQS Design Pattern 59 m1m2 m1m2 m1m2 SQS Q1 SQS Q2 SNS T1 Reliable Multi Push Reliable Message Delivery
- 60. SNS + SQS 60 Producer Producer Producer m1m2 Consumer Consumer Consumer DB m1 m1m2 m1m2 SQS Q1 SQS Q2 SNS T1 Consumer Consumer Consumer ES m1
- 61. Batch Pipeline Architecture Putting the Pieces Together 61
- 62. But First …. 62
- 63. What Does Agari Do? 63
- 64. What Does Agari Do? 64 Customers email metadata apply trust models email + trust score Agari’s Current Product
- 65. What Does Agari Do? 65 Enterprise Customers email metadata apply trust models email md + trust score Agari’s Future Product
- 66. Batch Pipeline Architecture Putting the Pieces Together 66
- 67. Batch Architecture 67 •S3 to hold all source & computed data (Avro) •EMR Spark for scoring + summarization •Apache Airflow for hourly job scheduling •SNS+SQS for messaging •ASG Importer to import •WebApp in Ruby-on- Rails
- 68. Tackling Cost & Timeliness Leveraging the AWS Cloud 68
- 69. Tackling Cost 69 Between Hourly Runs During Hourly Runs
- 70. Tackling Timeliness Auto Scaling Group (ASG) 70
- 71. ASG - Overview 71 What is it? A means to automatically scale out/in clusters to handle variable load/traffic A means to keep a cluster/service of a fixed size always up
- 72. ASG - Data Pipeline 72 importer importer importer importer Importer ASG scaleout/in SQS DB
- 73. 73 Sent CPU ACKd/Recvd CPU-based auto-scaling is good at scaling in/out to keep the average CPU constant ASG : CPU-based
- 74. ASG : CPU-based 74 Sent CPU Recv Premature Scale-in Premature Scale-in: • The CPU drops to noise-levels before all messages are consumed • This causes scale in to occur while the last few messages are still being committed
- 75. 75 Scale-out: When Visible Messages > 0 (a.k.a. when queue depth > 0) Scale-in: When Invisible Messages = 0 (a.k.a. when the last in-flight message is ACK’d) This causes the ASG to grow This causes the ASG to shrink ASG : Queue-based
- 76. 76 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • ASG • EMR Spark • ASG • EMR Spark
- 77. Tackling Operability & Correctness Leveraging Tooling 77
- 78. 78 A simple way to author and manage workflows Provides visual insight into the state & performance of workflow runs Integrates with our alerting and monitoring tools Tackling Operability : Requirements
- 79. Apache Airflow Workflow Automation & Scheduling 79
- 80. 80 Airflow: Author DAGs in Python! No need to bundle many config files! Apache Airflow - Authoring DAGs
- 81. 81 Airflow: Visualizing a DAG Apache Airflow - Authoring DAGs
- 82. 82 Airflow: It’s easy to manage multiple DAGs Apache Airflow - Managing DAGs
- 83. Apache Airflow - Perf. Insights 83 Airflow: Gantt chart view reveals the slowest tasks for a run!
- 84. 84 Apache Airflow - Perf. Insights Airflow: Task Duration chart view show task completion time trends!
- 85. 85 Airflow: …And easy to integrate with Ops tools! Apache Airflow - Alerting
- 86. 86 Apache Airflow - Correctness
- 87. 87 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 88. Near-Real Time Data Pipelines Stream Processing @ Agari 88
- 91. 91 The Architecture is composed of repeated patterns of : ASG-based compute Kinesis streams (i.e. AWS’ managed “Kafka”) Lambda-based Avro Schema Registry NRT Architecture
- 92. Avro Schema Registry Avro Schema Storage 92
- 93. 93 { "namespace":"com.agari.ep.collector.model", "type":"record", "doc":"This Schema describes the server-side configuration of Agari's Enterprise-Protect Collector", "name":"collector_config", "fields":[ {"name": "email_log_enabled", "type": "boolean"}, {"name": "email_log_interval_seconds", "type": ["int", "null"]}, {"name": "email_log_bucket_name", "type": "string"}, {"name": "phone_home_interval_seconds", "type": "int"}, {"name": "phone_home_sns_topic_ARN", "type": "string"}, {"name": "config_pull_interval_seconds", "type": "int"}, {"name": "receiver_netblocks", "type": ["null", {"type": "array", "items": "string"}]}, { "name": "connecting_ip", "type": [ "null", { "type": "record", "name": "connecting_ip_record", "fields": [ { "name": "received_header_index", "type": "int" }, ] …… A self-describing (schema’d) serialization format What is Avro?
- 94. What is Avro? 94 Typically, the schema is stored in the same file as the data it represents In HDFS, where files are typically large, the schema overhead is negligible
- 95. Schema Registry 95 P P C C C Kinesis Stream In streaming, where each record may be sent individually, the schema will be the majority of the data transmitted! This is a fat message Can we be smarter?
- 97. What is AWS Lambda? 97 AWS-hosted code execution environment (Python, Node, Java, Ruby) You upload some code & specify a simple memory and CPU profile (e.g. medium CPU, 256 GB memory) The code will get a new version (e.g. v2) Code Rollback as easy as setting $LATEST alias to a previous version (e.g. $LATEST=v1)
- 98. Elastic Stream Processing How Do We Handle Increasing Traffic? 98
- 100. Elastic Stream Processing 100 P P C C C Kinesis Stream
- 101. Elastic Stream Processing 101 C C C Kinesis Stream C C C P P Agari Scaling Utils
- 102. Open Source Plans 102 In late Q2/early Q3, we plan to open-source our cloud tools for : • Avro Schema Registry & • Agari (Kinesis+ASG) scaling tools To be notified, follow @AgariEng & @r39132
- 103. Acknowledgments 103 • Vidur Apparao • Stephen Cattaneo • Jon Chase • Andrew Flury • William Forrester • Chris Haag • Mike Jones • Scot Kennedy • Thede Loder • Paul Lorence • Kevin Mandich • Gabriel Ortiz • Jacob Rideout • Josh Yang None of this work would be possible behind the contributions of the strong team below