This document discusses applying chaos engineering principles to complex data systems. It recommends defining a steady state, acknowledging real-world events, running manual experiments in production, and automating production experiments. This helps discover weaknesses and manage the data lifecycle through stages like development, experimentation, deployment, and production. It also presents the idea of using a "Git for Data" system like lakeFS to version data and enable features like branching, rolling back, and atomic updates to more easily develop, test, and recover from errors in data pipelines.

1. 1

2. “Don’t make the assumption that things in
Spark just work, there is a good chance
that Spark underneath the hood is going to
do something unexpected”
2
Holden Karau
Apache Spark PMC
2017, BeeScala
@adipolak

3. What can go wrong with Data flows?
EVERYTHING
3
● new component logic
● new data source
● introducing incompatible schema
change
● Kafka broker failed to validate record
● Spark job runs twice, in parallel
● changing tables’ relationship keys
● accidentally delete yesterday’s
`events/` partition
● data duplication
@adipolak

4. Testing is hard!
Distributed data systems with many moving parts
4
Unit testing
Integration testing
System testing
E2E …
@adipolak

5. Chaos Engineering in the
World of Large-Scale Complex
Data Flow
Adi Polak
Treeverse

6. 4 Principles:
6
• Defining a steady-state
• Acknowledging a variety of real-world events
• Running manual experiments in production
• Automating production experiments
@adipolak
Chaos Engineering

7. #1 - Steady state
• system’s throughput
• error rates
• latency percentiles
• etc
DevOps / SRE
7
• Data products requirements
Data Engineer
@adipolak

8. Data Product Requirements
8
• Data Quality
• Accuracy
• No duplications
• SLA
@adipolak

9. In 3 lines of code..
9
# Read data from file
df = spark.read.parquet('s3a://bank_transactions/ts=123123123/’)
# Some transformation over the data related to finance
updated_df = df… do some magic!
updated_df.write.mode('append').parquet('s3a://bank_transactions/ts=123123123/’)
1
2
3
4
5
6
7
8
9
10
@adipolak
Data Duplication

10. #2 - Vary Real-world Events
hardware failures like :
• servers dying
• spike in traffic
DevOps / SRE
10
• schema change
• corrupted data record
• data variance
• accidentally delete yesterday’s `events/`
partition
Data Engineer
@adipolak

11. 11
@adipolak

12. #3 – Run Experiments in Production
• Production machines and network
DevOps / SRE
12
• Production data
Data Engineer
@adipolak
🙀

13. Experimental environment
On production data in production ..
13
s3://data-repo/collections/foo
s3://data-repo/exp1/collections/foo
@adipolak
★
★

14. #4 – Automate Experiments to Run
Continuously in Prod
• Discover weaknesses
DevOps / SRE
14
• Manage Data lifecycle in stages
Data Engineer
@adipolak
& Minimize Blast Radius

15. Roll Back
Troubleshoot
Production
Version control
Best Practices & Data
Quality
Deployment
Data product stages & requirements
Experimentation
Debug
Collaborate
.
Development
15
@adipolak

16. 16
What's the best
way to automate
data stages
propagation in
production?
@adipolak

17. Branching Strategy
Like source control -
17
@adipolak

18. 18
@adipolak
Git for Data

19. Challenge Solution
$ spark.read.parquet
(‘s3://my-repo/<commit_id>’)
$ lakectl revert main^1
$ lakectl branch create my-branch
$ lakectl merge my-branch
main
$ lakectl branch create new-logic
$ lakectl merge new-logic main
Quickly recover from an error
Develop in Isolation
Troubleshoot
Atomic Update
Reprocess
Git for Data

21. 21
How Does lakeFS Work?
@adipolak

22. 22
What Does A Typical Environment Look
Like?
@adipolak

23. 23
Recap
• Adopting Principles of Chaos Engineering to data systems
• Data Lifecycle Management stages
• Git for Data
• lakeFS