Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
In this talk, we'll explain our journey from having near-zero monitoring to having all of our infrastructure monitored with the necessary metrics and alerts. We will share with the audience some of the mistakes we did and what lessons we have learned. We currently have around 200 instances monitored with a comfortable cost-effective in-house monitoring stack based on Prometheus. We want to demonstrate that you don't need to have a big fleet to embrace Prometheus and that it is a non-expensive solution for monitoring. ---------- ShuttleCloud is a small startup specialized in email and contacts migrations. We developed a reliable migration platform in high availability used by clients like Gmail, GContacts and Comcast. For example, Gmail alone has imported data for 3 million users with our API and we process hundreds of terabytes every month. ------------- Follow us on Twitter: @ShuttleCloud: https://twitter.com/ShuttleCloud @ShuttleCloudEng: https://twitter.com/ShuttleCloudEng ShuttleCloud.com
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
Semelhante a Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016 (20)
Último
Último (20)
Prometheus Is Good for Your Small Startup - ShuttleCloud Corp. - 2016
- 1. Prometheus Is Good for Your Small Startup ShuttleCloud Corp. 2016
- 3. ShuttleCloud • Techstars 2011 • Chicago & Madrid • Email & contacts import API • ISPs and email & address book providers
- 4. ShuttleFacts • Gmail: 3 million users with our API • High availability HA SLA 99.5% • 6 TB/h • +18k migrations per day • ~30 million emails per day • ~3 million contacts per day • 247 providers around the world
- 5. What to expect from this presentation • Disclaimers • In the beginning… • A new dawn • Middle Ages • Modern History • Back to the Future
- 6. Disclaimers
- 10. In the beginning… (What we had)
- 12. Pingdom
- 16. A new dawn (New metric and alert systems)
- 18. Why Prometheus? • Metrics have labels - flexibility (can be added/changed) • No need of external services (i.e. Sensu with RabbitMQ) • Service Discovery from our DNS • Easy to install and test
- 19. Some first steps • Bronze Age: The targets were statically fed using Ansible • Iron Age: DNS service discovery • Operations Metrics from node_exporter • Others using textfile in node_exporter:
- 20. Some first alerts • Only Operations Alerts: • Hard drive usage • InstanceDown • Absolute thresholds HD 85% capacity — Send email HD 90% capacity — Page
- 22. Middle Ages (Business Metrics: Operation Exporter)
- 26. Push Gateway • Prometheus Pushgateway (https://github.com/prometheus/pushgateway) • Metrics Cache • Publisher frequency vs Prometheus scraping time Prometheus Publisher Push Gateway
- 27. Metrics • 3 main metrics: • incoming_operations_last_10_min* • incoming_status_last_10_min* • migrations_last_5_min • Gauges (and we were wrong!)
- 28. Gauges consequences • Alerts kept fired unnecessarily 10min t=now 10min t=now
- 30. HA • Publisher code is not ready to have two instances of itself • Publisher operations are not atomic Prometheus Publisher Push Gateway Prometheus Publisher
- 31. Clusterize • Clustering solution because of other services in the same ecosystem Prometheus Publisher Push Gateway Prometheus Publisher
- 33. Modern History (Business Metrics: Revamp)
- 34. Journey to the Past Prometheus Publisher Push Gateway
- 36. From Publisher to Exporter • Cron based script • Gauges • Aggregation in Publisher • Stateful Operation Exporter • Standalone App (Dockerized) • Counters • Aggregation in Prometheus • Prometheus handles resets • Stateless Prometheus Publisher
- 37. From Publisher to Exporter Prometheus Publisher Operation Exporter
- 38. HA • Operation Exporter is Stateless • Aggregation in Prometheus: • max(…) without(instance) Operation Exporter Operation Exporter
- 39. Metrics • 3 metrics: • operation_requests_total • operation_statuses_total • operation_errors_total • Counters (^_^) /
- 41. Alerts • Predict Linear vs Absolute Threshold
- 42. Alert Manager • 0.0.4 0.1.1(now 0.5) • Alerts as a condition-based tree vs condition-based list • Similar alerts are grouped when notified • Pagerduty integration improved
- 47. Our new fancy Dashboard
- 49. Blackbox Exporter • Metrics on certificates expiring date • Kudos to: (http://www.robustperception.io/get-alerted-before-your-ssl-certificates-expire/)
- 53. Approximate usage • Monitor 200 instances with: • 1 Prometheus instance in GCE (n1-standard-2) • 1 HD (used ~30GB with default retention period 15d) • 1 Meta-monitoring instance (f1-micro)
- 54. Back to the future (Next Steps)
- 55. CouchDB Exporter • Currently exporting metrics with textfile via node_exporter
- 57. Simplify current Alert System Super critical Project Normal Project Normal Project
- 58. Instrumenting Code • Currently: • Operation Metrics • Business Metrics (~Blackbox Monitoring) • Missing Metrics from Services