The industry is moving toward a microservices architecture, and many companies have embraced container orchestration solutions such as Kubernetes. DigitalOcean is no different. Over the past year, DigitalOcean’s Delivery team has been building a runtime platform based on Kubernetes with the goal of making shipping code easier. The system has empowered service owners to quickly and efficiently deploy and update their applications. A vital component is a white box monitoring and alerting solution based on Prometheus and Alertmanager. Sneha Inguva offers an overview of the system and shares problems encountered, potential solutions, and key lessons learned in the process. Sneha dives into the setup of Prometheus and Alertmanager that allows service owners to instrument their own metrics and alerts, explaining the service owner’s point of view and the internals that allow for the dynamic addition of alerts, and offers a glimpse of future modifications to the system. Join in to learn how to leverage open source tools for your monitoring and alerting needs.

1. @snehainguva

3. digitalocean.com
frustrating deployment and
update process

4. digitalocean.com
container orchestration
solutions with a bazillion
features

5. digitalocean.com
monitoring dynamically
changing services

6. observability in a dynamically
scheduled world

7. digitalocean.com
leveraging prometheus and
alertmanager for cluster monitoring
and alerting

8. digitalocean.com
about me
software engineer @DigitalOcean
former delivery, currently observability
kubernetes, prometheus

9. digitalocean.com
the plan:
● the “olden” days vs. container orchestration
● docc at DigitalOcean
● prometheus + alertmanager and docc
● alerting in action: examples
● potential pitfalls
● next steps

10. digitalocean.com
the “olden” days:
service owners write an application
provision a server with chef or ansible
use a CI/CD pipeline, bash scripts, or other tools
to deploy and update application on VM

11. digitalocean.com
the “olden” days:
use nagios + various plugins to monitor
use collectd + statsd + graphite/carbon

12. digitalocean.com
the “olden” days:
longer to provision than write actual service
hard to set up monitoring services
blackbox monitoring NOT insightful
whitebox monitoring services NOT easily queryable

13. digitalocean.com
docc: Digital Ocean Command Center
a tool for deploying containerized,
stateless applications

14. digitalocean.com
post-docc:
abstraction layer on top of kubernetes
deployments and updates take minutes, not hours
easy-to-use CLI

15. Source: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.png

16. CLI DOCCSERVER
daemonset → pods
service

17. digitalocean.com
post-docc:
service owners write an application
service owner dockerizes application
describe application in json manifest file
deploy!

18. digitalocean.com
post-docc:
view running applications
get application logs
easily scale, update, or restart applications

19. digitalocean.com
But what about monitoring?

20. digitalocean.com
Let’s use
prometheus + alertmanager

21. Source: https://github.com/prometheus/prometheus

22. digitalocean.com
Why use prometheus and
alertmanager?

23. digitalocean.com
easy to deploy

24. digitalocean.com
flexible and extensible

25. digitalocean.com
labelling works well with kubernetes primitives

26. digitalocean.com
complementary kubernetes service discovery

27. digitalocean.com
low-level metrics leveraged via
a strong query language

28. digitalocean.com
counters
gauges
histograms
summaries

29. digitalocean.com
alertmanager:
easily deployed alongside prometheus
dedupes alerts
high availability configuration
multiple receiver options

30. digitalocean.com
any downsides?
push vs. pull model
service owner must instrument application

31. digitalocean.com
putting it all together

32. digitalocean.com
instrument your application
use prometheus golang client
expose metrics endpoint

33. digitalocean.com
specify metrics, ports, alerts in your manifest file
Which metrics endpoint should be scraped?
Which container port needs to be exposed?
Specify alerting rule, duration interval, and channel.

34. digitalocean.com
use docc CLI to deploy your application
CLI DOCCSERVER
$ docc deploy manifest.json

35. digitalocean.com
daemonset → pods
service
promconfig
alertconfigalertmanager
docc

36. digitalocean.com
prometheus talks to the kubernetes api and grabs
the metrics endpoint and port information
promconfigservice

37. digitalocean.com
promconfig grabs alert information and
rewrites prometheus rules file
promconfigservice

38. digitalocean.com
alertconfig grabs alert routes and
rewrites alertmanager configuration file
service alertmanager alertconfig

39. digitalocean.com
Some stats

40. digitalocean.com
300+ production applications
1.5 million+ timeseries
100+ prometheus alerts

41. digitalocean.com
What should we monitor?

42. digitalocean.com
counters: cumulative, increasing metric
gauges: single metric that goes up or down
histograms: samples and buckets observations
summaries: also samples observations but can calculate
things like quantiles

43. digitalocean.com
latency - histogram + summaries
traffic - counters + rate()
error - counters + rate()
saturations - gauge

44. digitalocean.com
R - request rate
E - error rate
D - duration

45. digitalocean.com
U - utilization
S - saturation
E - error rate

46. digitalocean.com
cluster CPU reservation
node memory utilization
loadbalancer connection error rate
service http request duration
real-life examples

47. digitalocean.com
How should we alert?

48. digitalocean.com
State-based alerts
Is there a divergence between expected
state and actual state of a service?

49. digitalocean.com
State-based alerts
Is my service up and/or scrapeable?
absent(up{kubernetes_name="doccserver"}) or
sum(up{kubernetes_name="doccserver"}) == 0
Do I have the # of loadbalancers I expect?
sum(up{kubernetes_name="loadbalancer"}) < 3

50. digitalocean.com
Threshold alerts
Do any of our measured metrics exceed a lower or
upper bound?

51. digitalocean.com
Threshold alerts
Is our loadbalancer at 50% capacity in terms of sessions?
max(haproxy_frontend_current_sessions /
haproxy_frontend_limit_sessions) BY
(kubernetes_node_name, frontend) * 100 > 50
Are 50 percent of tests taking longer than 10 minutes?
max(test_duration_seconds{quantile="0.5",res
ult="pass"}) BY (test_name) > 600

52. digitalocean.com
Common pitfalls

53. digitalocean.com
Pitfall #1: Alerting fatigue

54. digitalocean.com
Solution: Slack and/or Pagerduty
send only the most urgent, production alerts to pagerduty
try out different promQL queries to have less spikey metrics

55. digitalocean.com
Solution: Dedupe and group alerts

56. digitalocean.com
Pitfall #2: Confused service owners

57. digitalocean.com
Solution: Docs and suggested alerts
extensive documentation and tutorials, accessible from CLI
prometheus slack channel for real-time help
standard alert examples

58. digitalocean.com
Pitfall #3: Who owns what?

59. digitalocean.com
Solution: opinionated manifest file
services owner must include maintainer information
aerts themselves include descriptions and summaries with
several labels
alerts must include team-specific receivers

60. digitalocean.com
Pitfall #4: Monitoring the monitors

61. digitalocean.com
Solution: Duplicate promethei and HA
alertmanager
alertmanager
alertmanager alertmanager

62. digitalocean.com
Solution: Deadman’s switch
ALERT JustKeepSwimming
IF vector(1)

63. digitalocean.com

64. digitalocean.com
#1: Automated alerts
utilize user-defined memory and cpu limits for
threshold alerts
automatic state-based alerts

65. digitalocean.com
#2: Leverage metrics for autopilot
user trusts in our custom controllers and
schedulers
collect metrics and build model about resource
usage over time
accordingly adjust limits and alerts

66. digitalocean.com
#3: Leverage metrics for autoscaling
services based on resource usage, #
connections, etc.
loadbalancers based on # of frontend and
backend connections
# of worker nodes based on memory and cpu
capacity metrics

67. digitalocean.com
a brave new world of container
orchestration
OSS whitebox monitoring
extensibility

68. sources
● The best prometheus tutorials you will ever
read, Julius Volz
● Actual Prometheus Website, Julien Friedman
● Kubernetes Project