Monitoring even a modestly-sized systems infrastructure quickly becomes untenable without automated alerting. For many metrics it is nontrivial to define ahead of time what constitutes “normal” versus “abnormal” values. This is especially true for metrics whose baseline value fluctuates over time. To make this problem more tractable, Datadog provides outlier detection functionality to automatically identify any host (or group of hosts) that is behaving abnormally compared to its peers. These slides cover the algorithms we use for outlier detection, and show how easy they are to implement using Python. This presentation also covers the lessons we've learned from using outlier detection on our own systems, along with some real-life examples on how to avoid false positives and negatives. Learn more at www.datadoghq.com.

1. One of these things is not like
the others
Automatically Detecting Outliers
Homin Lee, Data Scientist

2. Outline
● Monitoring
● Alerting
● Outlier vs. Anomaly Detection
● Outlier Detection Algorithms
● Our Python Implementation

3. Monitor Everything

4. Monitor Everything
Datadog gathers performance data from all your application components.

5. Monitor Everything

6. Monitor Everything

7. Monitor Everything?

8. Alerting

9. Alerting?

10. Alerting?

11. Outlier Detection

12. Outlier Detection

13. Outlier Detection

14. Outliers vs. Anomalies

15. Outlier Detection Algorithms
MAD
median absolute deviation
DBSCAN
density-based spatial clustering of applications with noise

16. Robust Outlier Detection Algorithms

17. Median Absolute Deviation
MAD(D) = median( { |di
- median(D)| } )

18. Median Absolute Deviation
MAD(D) = median( { |di
- median(D)| } )
D = { 1, 2, 3, 4, 5, 6, 100 }

19. Median Absolute Deviation
MAD(D) = median( { |di
- median(D)| } )
D = { 1, 2, 3, 4, 5, 6, 100 }
median = 4

20. Median Absolute Deviation
MAD(D) = median( { |di
- median(D)| } )
D = { 1, 2, 3, 4, 5, 6, 100 }
median = 4
deviations = { -3, -2, -1, 0, 1, 2, 96 }

21. Median Absolute Deviation
MAD(D) = median( { |di
- median(D)| } )
D = { 1, 2, 3, 4, 5, 6, 100 }
median = 4
deviations = { -3, -2, -1, 0, 1, 2, 96 }
abs deviations = { 0, 1, 1, 2, 2, 3, 96 }

22. Median Absolute Deviation
MAD(D) = median( { |di
- median(D)| } )
D = { 1, 2, 3, 4, 5, 6, 100 }
median = 4
deviations = { -3, -2, -1, 0, 1, 2, 96 }
abs deviations = { 0, 1, 1, 2, 2, 3, 96 }
MAD = 2

23. Median Absolute Deviation
MAD(D) = median( { |di
- median(D)| } )
D = { 1, 2, 3, 4, 5, 6, 100 }
median = 4
deviations = { -3, -2, -1, 0, 1, 2, 96 }
abs deviations = { 0, 1, 1, 2, 2, 3, 96 }
MAD = 2 (std dev = 33.8)

24. Median Absolute Deviation
Parameters: Tolerance, Pct
} tol. = 3.0

25. DBSCAN

26. DBSCAN
Parameters:
epsilon, min_samples

27. DBSCAN
1 dd/2d/4 3d/4

28. DBSCAN
1 dd/2d/4 3d/4

29. DBSCAN
1 dd/2d/4 3d/4
~ median(dist from median series) × tolerance

30. MAD or DBSCAN?

31. MAD or DBSCAN?

32. Some subtleties

33. Some subtleties

34. Some subtleties

35. Python
def MAD(slist, tol, pct):
val_array = np.concatenate(slist)
median = np.median(val_array)
diffs = np.abs(val_array - median)
mad = np.median(diffs)
outlier_factor = tol*mad / NORM_CONSTANT
outliers = []
for series in slist:
series_diffs = np.abs(series - median)
outlier_values = series_diffs[series_diffs > outlier_factor]
pct_outliers = 100 * (len(outlier_values) / float(len(series_values)))
if pct_outliers > pct:
outliers.append(series)
return outliers

36. Python
def DBSCAN(slist, tol):
median_series = np.median(slist, axis=0)
dists = scipy.spatial.distance.cdist(values_array,
np.array([median_series]))
eps = tolerance*np.median(dists)/NORM_CONSTANT
db_scan = sklearn.cluster.DBSCAN(min_samples=1, eps=eps)
db_labels = db_scan.fit_predict(values_array)
most = np.argmax(np.bincount(db_labels))
return [slist[i] for i, l in enumerate(db_labels) if l != most]

37. Thanks!

38. Appendix

39. DASHBOARDS
Build Real-Time Interactive Dashboards
CORRELATION
Search And Correlate Metrics And Events
See It All In One Place
Your Servers, Your Clouds, Your Metrics, Your Apps, Your team. Together.

40. COLLABORATION
Share What You Saw, Write What You Did
METRIC ALERTS
Get Alerted On Critical Issues
DEVELOPER API
Instrument Your Apps,
Write New Integrations
See It All In One Place
Your Servers, Your Clouds, Your Metrics, Your Apps, Your team. Together.

41. Flexible Pricing
To Match Your Dynamic Infrastructure.
Free
Up to 5 Hosts
1 Day retention
Custom metrics and events
Discussion group supported
Pro
Up to 500 Hosts
$15 Per Host / Month
13 Month retention
Custom metrics and events
Metric alerts*
Email supported
Enterprise
500+ Hosts
Contact us for pricing:
+1 866 329 4466
sales@datadoghq.com
Customized retention
Custom metrics and events
Metric alerts*
Email and phone supported