Dashboards are useless. Open YouTube if you want to watch something. What benefits could automation of streaming KPI metrics bring to your business, and what pitfalls and concerns are to be expected? From Time Series analysis approach to building distributed streaming data pipeline.

1. Never Fail Twice
How Playtech Mastered Failure Detection Across Distributed Systems

2. Bio
 Technical Architect with more than 18 y. of experience
 Passionate about IT
 Financial and Data Science background
 Last years in Research and Design projects

3. Agenda
• What is observability and monitoring
• Why this is hard
• Possible approaches
• How we solved it
• Future of the instrumentation and observability

4. Objectives
 Get in touch with time-series analysis
 Understanding Distributed Systems pro’s and con’s
 Understanding observability and instrumentation concepts

5. Observability
 Monitoring is for operating software/systems
 Instrumentation is for writing software
 Observability is for understanding systems
Charity Majors

6. Why is it difficult
 1. Various problems may lead to non-obvious system behaviour.
 2. Various metrics may have different correlations in time and space.
 3. Monitoring a complex application is a significant engineering endeavor in and of itself.
 4. There is a mix of different measurements and metrics.

7. System monitoring
in Playtech
 50+ multibranded sites, distributed all over
the world
 Multiple products
 Multichannel
 Different mix of integrations

8. On the shoulders of giants
A lot of companies
built their own
solutions for
monitoring their
systems.
There was not
always success
stories.

9. Etsy
 Etsy is a large online
marketplace of handmade
goods
 Their engineering team
collected more than 250,000
different metrics from their
servers
 They tried to find anomalies
using complex math
approaches.

10. Lessons
learnt from
KALE 1.0
Anomalies in other metrics should be used for root cause
analysis.
Alerts should only be sent out when anomalies are detected in
business and user metrics
A one-size-fits-all type of approach will probably not fit
at all
Anomaly detection is more than just outlier detection

11. Google SRE team’s BorgMon
 Google has trended toward simpler and faster monitoring
systems, with better tools for post hoc analysis
 [They] avoid “magic” systems that try to learn thresholds or
automatically detect causality
 Rules that generate alerts for humans should be simple to
understand and represent a clear failure
According to the authors of Site Reliability Engineering

12. Playtech
case
Past tool from HP is “one-fits-for-
all”
Low efficiency and side effects
False Positives and missed incidents
Horrible operability

13. Time Series
 A time series is a series of data points indexed (or listed or graphed) in time order
 Economical processes have a regular structure
 These are amount of sales in the shops, production of champagne, online transactions
 Usually they have seasonal periods and trend lines
 Using this information, simplifies analysis

14. Stationary Time-Series Data
 Is a stochastic process, which characteristics does not change
 White noise

15. Non Stationary Time Series
 Trend line
 Dispersion change

16. How to model that?
 Every measurement consists of a signal and an error
component/noise, because our processes are affected by many
factors
 Point_of_measurement = signal + error
 Subtract the model’s values from our measurements
 The more our model resembles the real signal, the more our
residue will approximate the error component or stationarity or
white noise

17. Example

18. Cut 30 min data piece

19. Regression or finding a trend line

20. Trend line subtracted
Looks like white noise

21. Dickey-Fuller test of an initial piece of data
Stationary hypothesis rejected

22. And after subtraction
Result is a stationary time series

23. Let’s take a moving average from our example

24. A bit of Salvador Dali

25. Compared with a next week data

26. Why Time Series DB matters
Optimized for handling time series data
No Updates. Facts do not change ever
Appending data only
Last data has been queried more often
InfluxDB is one of the best time series database

27. An
Important
Notice
The second level involves receiving such information and
making decisions as to whether they represent real problems
or outages.
This is the information consumption level.
The first level involves searching for anomalies in metrics and
sending out notifications if outliers are found.
This is the information emission level.

28. Overall
Architecture
 Python stack
 Built as a set of loosely
coupled components
 Executed on their own
Python virtual machines
 Event-driven design

29. Event Streamer
 Component that holds Workers, fetches data regularly, and tests this data against the statistical
models managed by Workers
 A Worker is the main working unit that holds a set of models together with meta-information
 Workers are fully independent and every cycle is executed using a threading pool

30. Rule Engine
 Consumes the information provided by the
Event Streamer
 Rules built as Abstract Syntax Tree
 Around 1500 matches per sec in one
process

31. We also measure dynamics
 We can take into account the speed and acceleration of the degradation of the metrics
 It correspond to, respectively, the severity and the predicted change in the severity of the incident
 Speed is an angular coefficient or a discrete derivative of a particular metric, which is calculated
for every violation
 The same applies to acceleration or the second order derivative

32. Some of our Rules examples

33. Model ensemble can be fine tuned

34. For every alert report is created

35. Alerta – open-source product for alerts
aggregation

36. Non Functional Moments

37. What future brings us

38. Q&A
 Thank you very much
 aleks.tavgen@gmail.com
 https://medium.com/@ATavgen/time-series-modelling-a9bf4f467687
 https://medium.com/@ATavgen/never-fail-twice-608147cb49b