An idea about how one could implement pluggable open source monitoring solution using Telegraf, Elastic, Grafana and Consul

1. PLUGABLE MONITORING
SALO SHP

2. Requirements
Scalable
Zero infrastructure downtime
Monitor everything, TRACE level
Support external HW equipment
Exhausting storage capacity

3. TELEGRAF
PLUGIN-DRIVEN AGENT FOR
COLLECTING & REPORTING
METRICS

4. Inputs
cpu
disk
kernel
mem
processes
swap
system
aerospike
apache
bcache
cassandra
ceph
cgroup
chrony
cloudwatch
conntrack
consul
couchbase
couchdb
disque
dmcache
dns_query
docker
dovecot
elasticsearch
exec
fail2ban
filestat
fluentd
graylog
haproxy
hddtemp
http_response
influxdb
internal
interrupts
ipmi_sensor
iptables
jolokia
kapacitor
kernel_vmstat
kubernetes
leofs
linux_sysctl_fs
lustre2
mailchimp
memcached
mesos
minecraft
mongodb
mysql
net
net_response
netstat
nginx
nginx_plus
nsq
nstat
ntpq
openldap
opensmtpd
passenger
phpfpm
ping
postgresql
postgresql_ext
ensible
powerdns
procstat
prometheus
puppetagent
rabbitmq
raindrops
redis
rethinkdb
riak
salesforce
sensors
snmp
solr
sqlserver
sysstat
tomcat
trig
twemproxy
varnish
zfs
zookeeper
amqp
http_listener
kafka
logparser
mqtt_consume
r
nats_consume
r
nsq_consumer
socket_listener
statsd
tail
tcp_listener
udp_listener
unbound
webhooks
zipkin

5. Aggregates
FOG computing
Net / HDD / CPU
Response time
Non-raw data
Outlier detection
Server impl. required
PROS CONS

6. Outputs
influxdb
amon
amqp
cloudwatch
cratedb
datadog
discard
elasticsearch
file
graphite
graylog
instrumental
kafka
kinesis
librato
mqtt
nats
nsq
opentsdb
prometheus_client
riemann
riemann_legacy
socket_writer

7. ELASTIC
DISTRIBUTED, RESTFUL
SEARCH AND ANALYTICS
ENGINE

8. For Metrics?!
Commonly used in many market verticals
Scalable
Shards
Replicas
Auto master detection and failover

9. YOU
PROJECT
DATE CLIENT
ANYTIME
KIBANA
DATA VISUALIZATION PLUGIN

10. GRAFANA
OPEN PLATFORM
FOR BEAUTIFUL
ANALYTICS

11. Datasources
Graphite
Elasticsearch
CloudWatch
InfluxDB
OpenTSDB
Prometheus
MySQL
Postgres

12. CONSUL
INFRASTRUCTURE
SERVICES DISCOVERY

13. Overview
1
2
N
http.consul.domain.local
:8600
:8600
:8600
:53
:53
:80
:80
http://172.27.72.27:80/btc

14. Watches
key - Watch a specific KV pair
keyprefix - Watch a prefix in the KV store
services - Watch the list of available services
nodes - Watch the list of nodes
service - Watch the instances of a service
checks - Watch the value of health checks
event - Watch for custom user events

15. Checks
Script <4KB
HTTP (curl)
TCP (netcat)
TTL from ext
Docker exec
{“check”: {
"id": "api",
"name": "HTTP API on port 5000",
"http": "https://host:5000/health",
"tls_skip_verify": false,
"method": "POST",
"header": {"x-foo":["bar", "baz"]},
"interval": "10s",
"timeout": “1s"}
}

16. Service configuration
{"service": {
"name" : "http",
"tags": [“sdev”,”rv51","infra12","aws"],
"port": 80,
"checks": [{
"id": "cwu-http",
"name": "Service 'http' health-check",
"script": "/usr/bin/cwu -s 80 ",
"interval": "30s"
}]}}
{"watches": [{
"type": "checks",
"state": "critical",
"handler":
"/usr/bin/cwu -o elastic"
}]}
function cwu_watch_socket() {
OUT=$(lsof -nP -i ${CWU_PROTO}:${CWU_SOCK})
RC=$?
cwu_evaluate_rc ${RC} "watch socket ‘${CWU_PROTO}:${CWU_SOCK}' -" "${OUT}"
}

17. OPENSOURCE TELASTIC DASHBOARD
HTTPS://GRAFANA.COM/DASHBOARDS/2597

18. ELASTALERT
PATTERNS ALERTING
FRAMEWORK

19. Rules
frequency - there are X events in Y time
spike - the rate of events increases or decreases
flatline - there are less than X events in Y time
{black,white}list - a certain field matches a list
any - any event matching a given filter
change - field has different values within some time

20. Alerts
Email
JIRA
OpsGenie
Commands
HipChat
MS Teams
Slack
Telegram
AWS SNS
VictorOps
PagerDuty
Exotel
Twilio
Gitter

21. What is actionable?
Supported rules
DIKWA Data layer 1
Elastic Data layer 2

22. QUESTIONS?