Anomaly Detection at Scale!
We all know how to create ML models, but the path to turning them into a highly scalable easy to use system by users is not always clear. What happens when you need to run thousands of them, on many different datasets, simultaneously and at a huge scale? AND, do it reliably so you can sleep well at night!! To achieve exactly that, we’ve decided to go down the serverless route and build an anomaly detection system on top of it. We’ll go over the pros and cons of building such a system using serverless and when such an approach could work for you. Our SpotLight anomaly detection system is capable of easily reusing ML models, and scale to run millions of time series simultaneously with ease. Our system eliminates manual work and allows our end users with no scientific background to set anomalies to detect in a plug and play way and get alerts in no time. In this talk, we’ll walk you through the architecture and share useful ideas you can adopt and implement in your own projects.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Anomaly Detection at Scale!
Semelhante a Anomaly Detection at Scale! (20)
Mais de Databricks
Mais de Databricks (20)
Último
Último (20)
Anomaly Detection at Scale!
- 1. Anomaly detection… at SCALE ! Opher Dubrovsky, BigData dev lead, Nielsen Max Peres, BigData engineer, Nielsen
- 2. The Problem with Data Photo by Pankaj Patel on Unsplash
- 3. Where is Waldo ? Photo by Pankaj Patel on Unsplash
- 5. The Problem with Data: SCALE !
- 6. Our Data Anomaly Photo by Jeff Kingma on Unsplash Anomaly Anomaly 1,131,219 time series
- 7. Agenda o Detecting anomalies o Architecting for cost and scale o Predicting the future o Models o Summary
- 8. Who are we ? Max Peres Big Data Engineer Motto The devil is in the details ! Opher Dubrovsky Big Data Dev Lead Motto The revolution is serverless !
- 9. Build marketing segments and device graphs Data is used for › Running campaigns › Business decisions Marketing Cloud
- 10. Nielsen Marketing Cloud Cloud native ~6,000 nodes/day ~60 TB/day 5 PetaByte ~6 million/day
- 11. Timeline 2019 2020 2021 Beta in production Scale improvements Scale redesign 15,000ts 35,000ts 300,000ts 500,000ts asked to alert on anomalies 146,189,940 data points Pilot created 1,100,000ts
- 12. Detecting Anomalies ? Actual weather 8:00 9:00 10:00 11:00 12:00 13:00 14:00 24 co 26 co 26 co 27 co 28 co 28 co 0 co Prediction Compare a prediction to the actual values! ANOMALY!
- 13. Naive Pilot - Relational Data Storage Size Query time Poc Beta Full scale Great for a pilot OK for a beta A bottleneck at scale
- 15. All Out on Serverless ! • Lambda Compute unit • Lazy execution • Low cost S3 storage Low Cost & Endless Scale
- 16. Architecture – Using Fan Out to Scale Work Manager Workers Calculate new data, predictions and anomalies Message Queue
- 17. Our Unit of Work TimeSeries: [1, 13, 15, 23, 42] Paths: [s3://….. , s3://….. , ] Message Aggregation period: 3 hours Model: ARIMA Configuration Postgres S3 Anomalies Only Time series Files Results
- 18. The Secret to Scale and Low Cost ? ▪ Low cost storage ▪ Low cost processing ▪ Endless scale
- 19. Monthly Costs 10,000 time series / month Cost Processing $2.70 Storage $0.000084 DB $1.66 TOTAL $4.36 Negligible ! • Single update run (100 time-series) - $0.0009
- 21. “Prediction is very difficult, especially about the future” Niels Bohr
- 22. Errors grow into the future Prediction Period 1 2 3 . . . . . N Acceptable Not Acceptable short predictions == small errors
- 23. Algorithm Flow 9 11 13 15 14 13 11 9 Training Data T T-1 T-2 T-3 . . . T-n 10 11 T+1 Prediction Time series 10 T+2 10 10 T+3 18 18 ANOMALY! Training Data Large Delta > 2.5 stdev (==5.4) Training Data stdev = 2.2
- 24. Life is Full of Difficult Decisions!
- 25. Supported Libraries Future Libraries
- 27. Anomaly
- 29. Choosing a model
- 30. Choosing a model
- 31. Choosing a model
- 32. ANOMALY!
- 33. A relationship will never work if there is no trust !
- 34. Building trust: 1 picture == 1000 words
- 35. Summary
- 36. Items we talked about 1. Keeping costs low 2. Designing for scale 3. The importance of trust 4. Prediction models
- 37. Our Recipe Low cost parts Scale - duplicate units Trust building features Models - support multiple
- 38. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.