Design a monitoring system with InfluxDB and the Tick stack.

1. Time Series Data
Tick Stack

2. Hello!
I am Gianluca Arbezzano
Software Engineer at CurrencyFair
@gianarb on Twitter and GitHub
2

3. 3
Open Source Maintainer and Contributors
penny.gianarb.it vim-php.org

4. 4
scaledocker.com
The book that drives you in production with
docker

5. “
“Try again. Fail again.
Fail better.”
cit. Samuel Beckett
5

6. Trust your system
To be familiar with your applications
you need to know what they are doing.
6

7. 1
Widespread
Monitoring Tools
tail -f /var/log/app.log
7

8. 8

9. 9
2016/04/15 15:42:46 [warn] 2330#0: *167 using uninitialized
variable,
client: 10.0.1.1, server: localhost.dev, request: "POST /auth
HTTP/1.1",
host: "localhost"
2016/04/15 15:44:44 [error] 2330#0: *171 FastCGI sent in stderr: "
PHP message: PHP Fatal error: Uncaught exception
'RuntimeException' with message 'All broken)[500]'
in /var/www/my/project.php:237
Stack trace:
#0 /var/www/index.php:45 ObjectService->flush()
#1 [internal function] ->save()

10. 2
Expensive to store
10

11. 2
Difficult to index
11

12. Difficult not impossible
They do an amazing work. And there are other tools!!
12

13. They are awesome for some use cases
▪ Extract informations
▪ They can be “human readable”
▪ … and others, and others
13

14. Keep your life amazing
Reduce your time series
to a timestamp
and a value (int or float)
14
This is cheap and useful

15. We are here to speak about
Time Series
[
{
"name": "log_lines",
"columns": ["time", "line"],
"point": [1400425947368, "here's some useful log info"]
}
]
15

16. EASY! EASY! EASY!
{
"name": "cpu_percent_use",
"columns": ["value"],
"point": 40
}
16

17. “
“Any intelligent fool can make things
bigger, more complex, and more violent.
It takes a touch of genius — and a lot of
courage to move in the opposite
direction.”
Cit. Ernest F. Shumacher
17

18. Time is a perfect sharding key
It means that Time Series scale really well
18

19. We have a set of tools to use
19

20. 20
InfluxDB
● Optimized to store time series data
● Open Source and easy to install (go
binary)
● Big community and ecosystem to
manage alert and collect metrics

21. 21
Easy
Install and start a go binary
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.0.0_amd64.deb
sudo dpkg -i influxdb_1.0.0_amd64.deb
Influxd -config /usr/local/etc/influxdb.conf

22. 22
Easy
HTTP API on port 8086
Support for UDP connections
Admin Panel on port 8083

23. 23
Easy
SELECT value
FROM cpu_load_short
WHERE region='us-west'

24. 24
T-Shirts time!

25. 25

26. 26

27. 27
[key] [fields] [timestamp]
temperature,machine=unit internal=3,external=10 1434055562000000035
Inline Protocol thinked to be smart and slim

28. 28
CorleyBenchmarksInfluxDBAdapterEvent
Method Name Iterations Average Time Ops/second
------------------------ ------------ -------------- -------------
sendDataUsingHttpAdapter: [1,000 ] [0.0026700308323] [374.52751]
sendDataUsingUdpAdapter : [1,000 ] [0.0000436344147] [22,917.69026]
UDP vs TCP protocol

29. 29
CREATE CONTINUOUS QUERY minnie ON world
BEGIN SELECT min(mouse) INTO min_mouse
FROM zoo GROUP BY time(30m) END
Continuous Query

30. Telegraf
https://github.com/influxdata/telegraf
Collector to grab and send
data from different sources to
InfluxDB and other databases
30
Based on Input and out Plugin System

31. 31
Telegraf Plugins

32. Kapacitor
https://github.com/influxdata/kapacitor
Trigger notifications and make
action in case of specific
behaviors
32
framework for processing, monitoring, and alerting on time series data

33. 33
Kapacitor high CPU alert
stream
|from()
.measurement('cpu_usage_idle')
.groupBy('host')
|window()
.period(1m)
.every(1m)
|mean('value')
|eval(lambda: 100.0 - "mean").as('used')
|alert()
.message('{{ .Level}}: {{ .Name }}/{{ index .Tags "host" }} has high cpu: {{ index .Fields "used" }}')
.warn(lambda: "used" > 70.0)
.crit(lambda: "used" > 85.0)
// Send alert to hander of choice.
// Slack
.slack()
.channel('#alerts')
// PagerDuty
.pagerDuty()

34. 34
Demo
https://github.com/gianarb/tick-php

35. 35
When you start to work with "micro"services
understand the topology of your connections is
really important
Time series can help you

36. 36
Use a dashboard to put together
different metrics and create not
obvious relations

37. 37
Why InfluxDB and not something else?
https://www.influxdata.com/influxdb-is-27x-faster-vs-mongodb-for-time-series-workloads/
27x greater write throughput
84x less disk space

38. 38
That’s it!
A series of great tools to monitor your
applications and your infrastructure

39. 39
A monitoring system isn’t for all

40. Thanks!
See you around
twitter.com/gianarb
40