At Databricks, we manage Apache Spark clusters for customers to run various production workloads. In this talk, we share our experiences in building a real-time monitoring system for thousands of Spark nodes, including the lessons we learned and the value we’ve seen from our efforts so far. The was part of the talk presented at #monitorSF Meetup held at Databricks HQ in SF.

1. Monitoring Large-Scale
Spark Clusters at
Databricks
Nanxi Kang
Apr.12th

2. 2
Who we are
Inventors of
• The largest open source project in data processing
• Unified platform for ETL, interactive query, machine learning, streaming, ...
Leader in Big Data ecosystem

3. 3
What we do
We offer a cloud-based Spark platform
• Demo shard

4. 4
Databricks platform
Databricks
Create
Configure
Terminate
Customer 1
Customer 2
ClusterClusterCluster
ClusterCluster
service
service
service
customer
requests

5. 5
Requirements of monitoring
As an engineer, I want to
• Understand the state of the service
• Have customer-oriented metrics (uptime, performance regression, ...)
• Be alerted on abnormality
• Ease debugging and troubleshooting

6. 6
Requirements of monitoring
As an engineer, I want to
• Understand the state of the service
• Have customer-oriented metrics (uptime, performance regression, ...)
• Be alerted on abnormality
• Ease debugging and troubleshooting
Technically
• Ingest metrics with low latency
• Aggregate metrics
• Scalability

7. 7
Ingest and aggregate metrics
Prometheus Grafana
Alert Manager +
PagerDuty
Databricks
service
service
Metrics
latency{operation=”runCommand”} = 60 alert if latency{operation=”runCommand”} > 100

8. 8
Ingest and aggregate metrics
Prometheus Grafana
Alert Manager +
PagerDuty
Databricks
service
service
Metrics
Kinesis

9. 9
Latency
Prometheus Grafana
Alert Manager +
PagerDuty
Databricks
service
service
Metrics
Kinesis
~ 30 seconds
~ 5 seconds
refresh every 1 ~ 5 mins
~ 5 seconds

10. 10
Recap Databricks platform
Databricks
Create
Configure
Terminate
Customer 1
Customer 2
ClusterClusterCluster
ClusterCluster
service
service
service
customer
requests

11. 11
Unique challenges
• Heterogeneous environments
• Scalability: large volume of metrics
– ~300K/min from Databricks
– ~2M/min from Customers
Customer 1
Databricks
ClusterClusterCluster
service
service
service
Prometheus

12. 12
Ingest metrics from customer env
Prometheus
Kinesis
Customer

13. 13
Ingest Metrics from Customer Env
Prometheus Grafana
Alert Manager +
PagerDuty
Kinesis
Customer
Kinesis2Prom

14. 14
What Next?
Let’s collect metrics!
Services can emit structured data about how they’re doing
• This is called whitebox monitoring
• Example: “how long does it take to finish a spark command?”
• Golden metrics (Latency, Traffic, Error, and Saturation)

15. 15
Whitebox monitoring
During a recent AWS S3 outage in us-east-1, our services
report failures to create Spark clusters.

16. 16
What’s missing?
• Hard to cover the end-to-end (cross-)service behaviors
• Whitebox metrics are great for causes, not symptoms

17. 17
What’s missing?
• Hard to cover the end-to-end (cross-)service behaviors
• Whitebox metrics are great for causes, not symptoms
Understand the user’s experience
• Use a fake user to record the service’s behavior
• This is called Blackbox monitoring
• Example: a user records the time from “he hits the command run UI
button” to “he sees the results/errors from UI.”

18. 18
Metrics in realtime

19. 19
Our monitoring pipeline needs to be monitored too!
Meta-monitoring
Prometheus Grafana
Alert Manager +
PagerDuty
Meta-Prometheus

20. 20
We can shard metrics and launch more Prometheus(es).
Scale out monitoring pipeline
Prometheus
Databricks
Prometheus
Kinesis2Prom
metric filters

21. 21
We can shard metrics and launch more Prometheus(es).
However, we cannot join metrics across partitions.
Scale out monitoring pipeline

22. 22
We can shard metrics and launch more Prometheus(es).
However, we cannot join metrics across partitions.
We are starting to offload extremely high-volume metrics to
Spark’s structured streaming.
Scale out monitoring pipeline

23. 23
Monitoring is useful!
- Since rollout, all teams at Databricks started using
dashboards and alerts
Add metrics as you build features
Building scalable and robust monitoring systems is hard
Take-aways

24. Thanks & We Are Hiring!
nkang@databricks.com