1.
Prometheus Day 2

2.
HELLO!I am Tom Tsai aks smalltown
- MaiCoin/AMIS SRE
- Taipei HashiCorp User
Group Organizer
- AWS User Group Taiwan
Staff
2

3.
3
http://bit.ly/taipei-hug

4.
Where is Your
4
?

5.
Outline
▫ Prometheus Basic
▫ Kubernetes Operator
▫ Prometheus Operator
▫ Thanos
▫ Internal/External Monitoring
5

6.
Prometheus Basic

7.
Prometheus Introduction
7
Exporter Prometheus Alertmanager

8.
Install Prometheus is Easy
▫ CM Tools: Ansible, Chef, Puppet, SaltStack
▫ Kubernetes Addons
▫ DigitalOcean
▫ ...
8

9.
Maintain Prometheus is Bothering (1/2)
9
▫ Problems Which Encounter Within K8S
- Configure Target for Many Services
- Configure Many Alert Rules
- Grafana Dashboard Management

10.
Maintain Prometheus is Bothering (2/2)
10
▫ Problem Which Encounter Everywhere
- How to Achieve HA
- Reliable Historical Data
- Downsampling

11.
Exclude All Problems Today!
11

12.
Kubernetes Operator

13.
Manage Production Service
13
Stateless Stateful
Setup
Scale Out/Up
Backup
Upgrade
Operation

14.
K8S Operator
14

15.
What is Kubernetes Operator
▫ An Operator is a Software with Specific Application
Knowledge
▫ Extends the Kubernetes through Custom
Controller/Resource
▫ Enable Users to Create, Configure, and Manage
Applications
15

16.
16
What is Kubernetes Operator
An Operator is a Domain Specific Controller

17.
17
Operation of Stateless Service
kubectl
ReplicaSet
app=web, env=prod
Pod
app=web
env=prod
scale up
desired=3 count=1
kubectl
ReplicaSet
app=web, env=prod
desired=3 count=3
Pod
app=web
env=prod

18.
Pod
app=web
env=prod
18
Operation of Stateful Service (Operator)
backup
kubectl
etcd Operator
app=3.0.13
size=3 backup-tool
interval=300s

19.
Prometheus Operator

20.
Maintain Prometheus is Bothering
20
▫ Problems Which Encounter Within K8S
- Configure Target for Many Services
- Configure Many Alert Rules
- Grafana Dashboard Management

21.
Prometheus Operator Overview
21
K8S Service
(Exporter)
Prometheus
Alertmanager
GrafanaK8S CRD:
ServiceMonitor
PrometheusRule

22.
22
K8S CRD Within Prometheus Operator (1/2)
▫ Prometheus
- Manage Prometheus StatefulSet
- Prometheus Configuration Reloader
- Prometheus Rules Reloader
▫ Alertmanager
- Manage AlertManager StatefulSet
- Configuration Reloader

23.
23
K8S CRD Within Prometheus Operator (2/2)
▫ ServiceMonitor
- 1 ServiceMonitor Mapping to 1 Exporter Service
- All ServiceMonitor Compose Prometheus
Configuration
▫ PrometheusRule
- 1 PrometheusRule Mapping to 1 Rule File

24.
24
How to Monitor A New Service
▫ Precondition
- Prometheus Install/Manage By Operator
▫ Add New Exporter and Related Service
▫ Add New ServiceMonitor/PrometheusRule
▫ Use Helm to Deploy, and The Changes Will be Reloaded
Automatically

25.
25
How to Add A New Grafana Dashboard
▫ Cook Dashboard
▫ Export Dashboard Json File
▫ Use Helm to Deploy to Related K8S ConfigMap
▫ Changes Trigger Grafana Watcher Sidecar to Reload
Automatically

26.
Maintain Prometheus is Bothering (1/2)
26
▫ Problems Which Encounter Within K8S
- Configure Target for Many Services
- Configure Many Alert Rules
- Grafana Dashboard Management
▫ Everything Uses Helm to Deploy and Store in Version
Control

27.
Thanos

28.
Maintain Prometheus is Bothering
28
▫ Problem Which Encounter Everywhere
- How to Achieve HA
- Reliable Historical Data
- Downsampling

29.
Thanos Main Components
▫ Sidecar
▫ Querier
▫ Store
▫ Compactor
29

30.
30
Global View
Disk
Sidecar Prometheus
Disk
Sidecar Prometheus
Deduplicate
Marge
Querier
...
Thanos Store API
Prometheus HTTP API

31.
31
Unlimited Retention (1/2)
Disk
SidecarPrometheus
Blocks
Blocks
Object Storage
Blocks

32.
32
Unlimited Retention (2/2)
Object Storage
Blocks
Querier
Store
Cache
Blocks
Thanos Store API

33.
33
Compaction & Downsampling
Object Storage
Blocks
Compactor
Disk
Blocks Blocks
Blocks

34.
34
Thanos Overview
Object
Storage
Compactor
Store Querier
Disk
Sidecar Prometheus
Disk
Sidecar Prometheus
...5
1 1
2
3
4

35.
Maintain Prometheus is Bothering
35
▫ Problem Which Encounter Everywhere
- How to Achieve HA
- Reliable Historical Data
- Downsampling

36.
Prometheus
Operator Support
Thanos
36

37.
37
Current Status
...

38.
Internal/External
Monitoring

39.
A Stone's Throw Away
▫ Who Monitor Your Monitor (Prometheus)
▫ Multi-Region Monitoring
▫ Some More Complicated Monitor
▫ ...
39

40.
40
The 80/20 Rule
▫ Pay Some Money to Use External Monitor to Avoid
Blind Spot
▫ E.g. Site 24X7, Monitis...etc

41.
41
THANKS!
Any questions?
You can find me at
▫ facebook.com/smalltown0110

42.
42
http://bit.ly/maiamis