The document discusses Marian Marinov's experience monitoring various systems and infrastructure. He aims to have a single solution for log and metrics collection but ends up with multiple Grafana dashboards and different log collectors. Collectd was found to be the easiest to set up and provides the most out-of-the-box metrics, while solutions like Elasticsearch and Kibana require too many resources for smaller setups. There is no single solution that can monitor everything.

1. Следете актуалните обяви за DevOps
Партньори:
Monitoring & Logging
Marian Marinov
mm@yuhu.biz

2. Следете актуалните обяви за DevOps
Партньори:
Who am I?
● Director of Engineering at Web Hosting Canada
● Former partner and Head of DevOps at SiteGround
● A SysAdmin and System Architect

3. Следете актуалните обяви за DevOps
Партньори:
What I have to monitor?
● 13 physical linux machines
○ Storage capacity (df/df -i)
○ S.M.A.R.T. of the drives
○ RAID (HW or Soft)
○ Network (routes, traffic and usage)
○ Performance (CPU, Mem, I/O, Processes)
○ Kernel logs
○ Service logs

4. Следете актуалните обяви за DevOps
Партньори:
What I have to monitor?
● 1 UPS
● 2 APC PUDs
● 2 Switches (SNMP statistics)
● 2 Thermostat (traffic, temp, humidity)
● 40+ LXC containers
○ Performance (CPU, Mem, I/O, Processes)
○ Storage capacity (df/df -i)
○ Service logs
● 2-3 Wifi access points
○ number of attached devices
○ traffic per-device

5. Следете актуалните обяви за DevOps
Партньори:
What I have to monitor?
● A few things for which I want traffic and power on time
○ 3 TVs
○ 3 Amplifiers
○ 4 Cameras
○ 1 Washing machine
○ 1 Dryer

6. Следете актуалните обяви за DevOps
Партньори:
What I wanted
● Single solution for log and metrics collection
● Single central interface

7. Следете актуалните обяви за DevOps
Партньори:
What I ended up having
● multiple grafana dashboards
● monitor events, instead of reading logs
● a bunch of different log collectors

8. Следете актуалните обяви за DevOps
Партньори:
What tested
● syslog-ng
● rsyslog
● Filebeat
● Prometheus node_exporter
● Loki
● Fluentd
● Clolectd
● StatsD
● Graylog
● PostgreSQL+timescale
● Grafana

9. Следете актуалните обяви за DevOps
Партньори:
Conclusions
● there is no one solution to rule them all
● SNMP is still the king for networking
● too many logging formats and DSLs

10. Следете актуалните обяви за DevOps
Партньори:
Conclusions
● there is no one solution to rule them all
● SNMP is still the king for networking
● too many logging formats and DSLs
● collectd was the easiest
○ with the most metrics out-of-the-box

11. Следете актуалните обяви за DevOps
Партньори:
Conclusions
● there is no one solution to rule them all
● SNMP is still the king for networking
● too many logging formats and DSLs
● collectd was the easiest
○ with the most metrics out-of-the-box
● ElasticSearch + Kibana require too much resources
○ Not usable for smaller setups
● Graylog uses a lot of CPU for the work it does
○ alerts can be based on number of events instead of parsing logs

12. Следете актуалните обяви за DevOps
Партньори:
Installation / Setup
● basic apt-get:
○ rsyslogd, syslog-ng, fluentd, collectd, filebeat, loki, node_exporter
○ statsd wanted full npm

13. Следете актуалните обяви за DevOps
Партньори:
Pros and Cons
● Syslog pros
○ can easily ingest netconsole kernel logging
○ very good performance
○ well documented and standardized interface
● Syslog cons
○ fire and forget
○ the syslog protocol
○ not enough parsing flexibility
○ syslog-ng was heavier then rsyslogd

14. Следете актуалните обяви за DevOps
Партньори:
Pros and Cons
● Loki/Node_exporter/filebeat/fluentd
○ very good parsing capabilities
○ filebeat was the easiest for me
○ reliable log delivery
○ different integrations
○ ready made grafana dashboards
● Loki/Node_exporter/filebeat/fluentd
○ very heavy on CPU
○ Loki did not have sysv init script :)

15. Следете актуалните обяви за DevOps
Партньори:
Interesting
● OAIEvals Collector - by Nikolay Stankov

16. Следете актуалните обяви за DevOps
Партньори:
DB integrations
1. Prometheus node-exporter
2. Fluentd
3. filebeat
4. syslog

17. Следете актуалните обяви за DevOps
Партньори:
Not out of the box
● Custom local collectors still have to go directly to your metrics DB
● Having a producer/subscriber greatly reduces the performance hit
● Fluent and fliebeat were the only one supporting kafka out of the box
○ https://github.com/hikhvar/mqtt2prometheus
○ https://github.com/toyokazu/fluent-plugin-mqtt-io

18. Thank you!
СЛЕДВАЩО СЪБИТИЕ
Лектор Дата Език
Следете актуалните обяви за DevOps
Партньори:
Monitoring & Logging
Marian Marinov 19.Mar.2024 Български
Contacts:
Marian Marinov
Github profile
Facebook profile

19. Следете актуалните обяви за DevOps
Партньори:
What do I have on the containers?
● NextCloud
● Home Assistant
● Mirrors
● VPNs
● NetBox
● Monitoring (Grafana, StatPing)
● Games (Minecraft, CS, PVPGN)
● IRC (server, bouncers, bots)
● Matrix, Mattermost
● Backups
● Streaming (FOSDEM streamer setup)
● DBs (PostgreSQL, MySQL, Redis, DragonFly, Timescale, InfluxDB, Mongo)
● Vitess, ProxySQL
● MPI (Gearman, MQTT, Kafka, RabbitMQ)
● Web stuff - Wiki, HAproxy, Nginx, Varnish
● OpenShift, OpenStack, K8s on VMs and physical
● A lot of other experiments

20. Следете актуалните обяви за DevOps
Партньори:
What storage do I use?
● Local + LVM
● DRBD+OCFS2
● iSCSI
● cLVM + iSCSI
● GlusterFS
● OrangeFS
● I had in the past:
○ Ceph
○ NFS
○ cLVM + ATAoE
○ cLVM + NBD