This document summarizes a presentation about how Netflix uses machine learning to detect issues in its operations. It discusses the challenges of applying ML to operational issues due to limited failure examples and dynamic environments. The presentation outlines Netflix's approach to real-time and near real-time anomaly detection using simple statistical algorithms like median-based detection, Hoeffding bounds, and Jensen-Shannon divergence to detect anomalies in scalar signals. For near real-time signals, it uses correlation and Mann-Whitney U tests on sliding windows to identify significant changes from a baseline. Visualization and communication of anomalies on Slack is also discussed.

1. How Computers Help
Humans Root Cause
Issues at Netflix
SETH KATZ
QCON NEW YORK, 2018

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-ml-root-case

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. ! Seth Katz
! 5 years at Netflix
! Focused on improving Netflix
operations
! Share what we’ve learned on applying
machine intelligence to operations
Hello!

5. I got paged!

6. Funny Tweet - Serious Situation

7. Agenda
! Netflix operations
! Approach and challenges to ML in operations
! Anomaly detection
○ Real-time
○ Near real-time
! Visualization and making it practical
! Reflections and takeaways

8. Android devices that can’t play a movie
exceeds 1%
What if we get this page?

9. Microservices
NQ NRDJS
Play API
manifest
Zuul

10. Android
NQ NRDJS
Zuul
Play API

11. Slack Message

12. Why is diagnosing pages hard
It’s 3am in the morning - are you thinking clearly?
Maybe you understand your microservice?
What about all the other services involved?
What about their push schedules in every region?

13. Hard problem - how to
build a minimum
viable product ?

14. Simple, Principled, Robust Anomaly Detection
Principled algorithms have guarantees you can use to reason
about for any data pattern
Simple algorithms that are very easy to implement. Don’t need
major frameworks, GPUs, Python, etc.
Approach and Challenges for ML

15. Wouldn’t be great if ...

17. Golden Age of AI
Approach and Challenges for ML

18. Why do Star Trek robots seem near, but
Lost In Space robots seem further into
the future

19. AI challenges in operations
Limited examples of outages
Cause and effect
Tribal knowledge

20. More AI challenges
Curse of dimensionality
Rapidly changing ground truth
Generalization to new problems

21. So what can we do? -
Real-time root cause
detection

22. Root cause for the oracle
Real Time Root Cause Detection

23. Real world example
Timeline
! 11:50:15 - Region failover from us-east-1 -> eu-west-1
! 11:51:12 - Service A timeouts increase 243% in eu-west-1
! 11:51:14 - Android movie errors increase 840%
Complete picture of what happens - time suggests causality

24. Victory?
We can only do this on metric
subsets
! Signals usually relatively stable
and slow changing
! Signal with up to date event
source
! Signals with rapid updates,
many samples.

25. How can we detect
scalar anomalies?

26. ! Anomaly very clear to
humans
! Limited data needed
! Historical trend
unnecessary
! Recovery also clear
! Principled signal analysis
possible
Scalar Anomaly Signal
Android error rate

27. What’s normal?

28. Median on a Stream.
If Incoming > Median:
Median = Median + Alpha
Else:
Median = Median - Alpha
! Alpha can be adjusted if consecutively on one side
! Need rapid data updates for timely convergence.

29. What’s abnormal?

30. ! Is the next data point from the same distribution as sample?
! Can I guarantee it is the same distribution with a desired level of
confidence?
! Do I need to assume my data is normally distributed (aka
Gaussian)?
! Hoeffding Bound
Hoeffding Bound

31. ! n=sample size
! d=desired certainty, eg .01 for 99%
! r=sample range, ie (max -min)
Hoeffding Bound Very Simple

32. Not Anomaly
Anomaly

33. Another problem -
detecting a bad config
push?

34. Consecutive histogram snapshots
11:10:15 11:10:20
Sharp
drop in
English
titles

35. Is there principled
way to measure
difference between
histograms?

36. Information Theory

37. Fair Coin
Entropy - Average Information
9-1 Biased Coin

38. How much entropy do
we lose if we estimate
histogram with wrong
probability
distribution?

39. Uniform Distribution Info
Loss

40. Minor Formula Change for Entropy difference
! Entropy
KL Divergence
! KL Divergence

41. Is KL divergence a
good score?

42. ! Not symmetric?
○ Take KL divergence in both directions and add
! No upper limit?
○ Normalize it
Jensen Shannon Divergence (JSD)

43. Real Time Root Cause Detection
Not Anomaly
Anomaly

44. Real time Algo Recap
Scalar?
Median for
expected
Hoeffding
Threshold
Yes?
JSD
Threshold?
Normalize
to 1
No?

45. How to communicate
anomalies?

46. ! Android movie errors increase 840%?
○ Increased from what?
○ Why not use z-score (number of standard deviations from
mean)?
Example

47. This is your brain on Pager Duty

48. Intuitive messages beat
mathematically precise ones

49. What about nearly
real-time signals?

50. More Time and More Data

51. Diurnal Patterns Prime Time
Night Time

52. ! Usually better for mean time to resolve than
mean time to detect
! Less precise timing
! Use correlation, but humans decide cause vs
effect
Drawbacks

53. Suspicious
Things

54. ! Is there an attribute over represented for sessions with 1234
error code?
○ Device?
○ UI version?
! Baseline Essential
○ What if only one UI version actually produces error code
1234?
Error Code 1234 is High?

55. How do we identify
significant change
from baseline?

56. Error 1234 UI Version 0.0.1
BaseLine 1000000 10000
Now 100000 1150
Two-Way Contingency Table
Use Chi-Squared test

57. Contingency Tables Fail
! Yes/No are past and
present the same
! Chi-squared says
significant, 99.999%
confidence
! Netflix is always
changing

58. Bonferonni’s principle
Are we there yet?
Eventually right by chance
if you ask enough!
Near real time signals

59. ! Contingency tables don’t work
! Convert it to a time series problem
Getting Correlation Right

60. Why would time
series work when
contingency tables
fail?

61. ! Chi-squared test is so sensitive because of very large samples
! Number of time windows much smaller - significance tests work
on smaller sets
Sensitivity

62. Correlation Windows
Time Window Pearson Correlation Score Error 1234 and UI Version 0.0.1
10am-10:30am .18
10:30-11:00am .2
11-11:30am .25
11:30am-12pm .95
Near real time signals

63. ! Mann-Whitney U Test on correlation values. (not Student’s t-test)
○ No Gaussian assumption involved
! Works best after human determines present is “interesting”
○ Eg, run after an alert fires
Significant Change?

64. Anomaly detection for
near real-time

65. InterQuartile Range
IQR = 20
Anomaly >
75% + N*IQR

66. Near real-time anomalies
2-3 am IQR Threshold
3-4 am IQR Threshold
Signal

67. Placeholder for dense graphs
! Microservices, cal pattern
! Color coded errors
! Sentence for more context
! Need to de-noise for slack to work well
! Need deduplication

68. Displaying anomalies
in context

69. Android
NQ NRDJS
Zuul
Play API

70. Visualization and making it practical

71. Summary on Slack

72. Reflections and
Takeaways

73. ! Scikit Learn and Tensorflow might be
overkill, at least for these algorithms
! Human curation reduces scope so we don’t
need a Danger Will Robinson intelligence
Back to basics - simple statistics
Reflections and Takeaways

74. Real time vs Near real time
! Timing suggests causality
! Useful for mean time to detect
! Careful choice of metrics needed
! Cause requires correlation
! Humans assign cause and effect
! More granular metrics
! Useful for mean time to resolve
! Diurnal pattern improved
predictions
Reflections and Takeaways
Real time Near real time

75. Get correlation right
! Contingency tables don’t work
! Correlation and Mann-Whitney U test works pretty well

76. A Summary Incident Approach
Android errors increased 850 percent?
Statistics +
Visualization
U-test
Hoeffding JSD Mann-WhitneyIQR Hourly

77. More Information, Q&A
Team
https://medium.com/netflix-techblog/lessons-from-building-observability-tools-at-
netflix-7cfafed6ab17
Me
https://www.linkedin.com/in/katzseth22202

78. Thank you.

79. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-ml-root-case