O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Monitor your CentOS stack with Prometheus

Carregando em…3

Confira estes a seguir

1 de 58 Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Monitor your CentOS stack with Prometheus (20)


Mais de Julien Pivotto (20)

Mais recentes (20)


Monitor your CentOS stack with Prometheus

  1. 1. Monitor your CentOS stack with Prometheus Julien Pivotto (@roidelapluie) CERN February 20st, 2019
  2. 2. user{name="roidelapluie"} 1 I like Open Source I like monitoring I like automation ... and all of that is my daily job at inuits
  3. 3. Traditional sysadmin-style monitoring Disk is full No zombie processes not 100 user connected ssh works swap is empty
  4. 4. Questions that come after: It's up but it is performant? It's down but for everyone? Its is degraded but are the users impacted? Is it even relevant?
  5. 5. Metrics Monitoring e.g. traditionally graphite Gather fine grained data at frequent interval Make them useful by labelling them ; store them Analyze them to understand what is going on
  6. 6. Metrics ARE PART OF monitoring Do not maintain a metrics + a "traditional monitoring" stack Alert from metrics directly!
  7. 7. Monitoring source Metrics then Logs then Spans (opentracing)
  8. 8. We are in the cloud era. Here are some buzzwords for you cloud, API, openstack, devops, docker, bimodal, stateless, kubernetes, orchestration, automation, serverless, docker, humanops, ansible, continuous deployment, cri-o, jenkins, agile, docker, red hat, containers, virtualization, provisionning, monitoring, observability...
  9. 9. What is the cloud Scale Velocity Change
  10. 10. On Premise looks like the cloud Nowadays you have no choice. Scale Velocity Change
  11. 11. What are the needs Automation Scalability
  12. 12. Bye bye all-in-one tools tools that don't scale tools you can not automate
  13. 13. We need deserve better tools Our customers ask us to respond fast, in seconds We make hundreds of operations per second What is your monitoring frequency... 5 minutes?
  14. 14. Time to get better tools / protocols
  15. 15. Prometheus https://prometheus.io/
  16. 16. Cloud Native Easy to configure, deploy, maintain Designed in multiple services Container ready Orchestration ready (dynamic config) Fuzziness
  17. 17. Data Centric A Metric in Prometheus has metadata: myql_global_status_handlers_total{handler="tmp_write"} 1122 And lots of function to filter, change, remove... those metadata while fetching them. => OpenMetrics.io
  18. 18. Open Source Apache 2.0 Go Support for multiple OS Many "exporters": https://github.com/prometheus/prometheus/wiki/Default- port-allocations
  19. 19. Simple 1 service = 1 things Takes care of its db (time based retention and/or disk space based retention)
  20. 20. RPM https://github.com/openshift/prometheus/blob/ master/golang-github-prometheus- prometheus.spec (I have not tested it) https://github.com/lest/prometheus-rpm
  21. 21. How does it work?
  22. 22. How does it work?
  23. 23. How does it work?
  24. 24. How does it work?
  25. 25. How does it work?
  26. 26. Exporters Exporters expose metrics with an HTTP API Bindings available for many languages Exporters do not save data ; they are not "proxies" and don't "cache" anything
  27. 27. Exploring Metrics
  28. 28. Exploring Metrics
  29. 29. Exploring Metrics
  30. 30. Exploring Metrics
  31. 31. PromQL mysql_global_status_commands_total
  32. 32. PromQL mysql_global_status_commands_total{command="select"}
  33. 33. PromQL mysql_global_status_commands_total {command=~"select|set_options"}
  34. 34. PromQL mysql_global_status_commands_total{command=~"select|se t_options"}
  35. 35. PromQL deriv(mysql_global_status_connections[5m])
  36. 36. PromQL {__name__=~".+innodb.+cache.*"} predict_linear(mysql_heartbeat_lag_seconds[5m], 60*2) sum(rate(mysql_global_status_commands_total{command=~" (commit|rollback)"}[5m])) without (command)
  37. 37. Demo prometheus node_exporter + file collector pushgateway within openshift
  38. 38. Pushing metrics Short lived batches can send metrics to a "pushgateway" Pushgateway is a project maintained by the prometheus maintainers
  39. 39. Alerting Prometheus has recording rules Record frequent queries or alerting Alert base on any PromQL queries.
  40. 40. One tool does one job... Prometheus collects data Exporters expose data Grafana graphes data Alertmanager dispatches alerts
  41. 41. Alertmanager Open Source Same developers as Prometheus Second service
  42. 42. What is the Alertmanager doing? Receives alerts Group them Inhibits them Dispatches them Deals with HA
  43. 43. How to alerts? Email Some vendors: Slack, Hipchat, VictorOps, pagerduty, ... Generic Webhook -> Plug in anything you want
  44. 44. High Availability 2 prometheus servers do exact the same job They send alerts to Alertmanagers Alertmanagers are clustered not to send the same notification twice
  45. 45. Grafana Open Source (Apache 2.0) Web app Specialized in visualization Pluggable Multiple datasources: prometheus, graphite, influxdb... Has an API!
  46. 46. History of Grafana Grafana is a fork of Kibana 3 ; used to be JS- Driven. Now fully featured, requires a database, multi- projects/users support, etc...
  47. 47. Grafana and Prometheus Prometheus shipped its own consoles Now it recommends Grafana and deprecated its own consoles
  48. 48. Grafana Dashboards
  49. 49. Grafana Dashboards
  50. 50. Time Picker
  51. 51. Configure Prometheus in Grafana
  52. 52. Configure Prometheus in Grafana
  53. 53. Prometheus Dashboard
  54. 54. Creating Grafana Dashboards Takes time Requires deep knowledge of the tools Improved over time Easy to share (json + online library) Try grafonnet-lib!
  55. 55. Conclusion Lots of data that can be explored in many ways (subqueries are coming) Trends and deviations are easy to calculate Can monitor both business and technical Very convenient to monitor any kind of stack
  56. 56. Conclusion Open Source Multiple small pieces Rich ecosystem For Cloud and On Premise
  57. 57. Julien Pivotto roidelapluie roidelapluie@inuits.eu Inuits https://inuits.eu info@inuits.eu Contact I will give a prometheus workshop at OSMC.de 2019!