Our goal is to eliminate alerts and a common mantra is “alert only if absolutely needed”. But that’s only for the monitoring-related people, whereas it seems as if the mantra for others is “whatever”. Excessive alerts are like Stephen King’s Langoliers – they eat our time. Decommissioned servers, flapping alerts, mysterious emails. Let’s bring on “Operation Forest” and see what – and even more importantly, how – can be done about that. The monitoring tool You choose is just a part of the solution. What good is your tool if nobody pays attention to it? We’ll talk about the usage of weapons like Terminology, Persistence and Involvement, as well as CommonSense and Policy as potential problem-solvers. Case studies (aka anecdotal examples) will be brought forward to illustrate ways alerts can become useless, be ignored and lead to serious outage. Also find out: what’s “Operation Forest”.

1. A day in oncall life
or
Operation Forest

2. No views are endorsed by employers,
past or present
Any resemblance to real-life entities is
coincidental

3. The storyteller
Involved with monitoring for 20 years
Worked at Zabbix for 6 years
OpenStreetMap mapper, tea drinker

4. Chapter 1: The Morning
on how I understood
that alerts are bad

5. 10:00 - This I know on a Friday
Alerts are good, because they inform us
Alerts are always useful

6. From: somebody@example.com
Subject: Re: High: appserver.example.com Important service down
thought it was short restart and it won't alert
--
Somebody

7. From: somebody@example.com
Subject: Re: High: oldappserver.example.com Server unreachable
This server has not been used for a year.
--
Somebody

8. From: somebody@example.com
Subject: Re: Average: zomgserver.example.com ZOMG workflow issue
ASDF and QWER workflow counts differ for last
ZOMG cycle. ASDF: 6762 QWER: 6760
Not much difference, hence ignoring
--
Somebody

9. Uncommon habits can bring
benefits in certain situations
We have to be careful with
declaring habits as bad

10. 13:00 – Habits = risks, time wasters
chmod 777
sudo su
"I'll keep an eye on this alert"

12. 14:13 – Alert fatigue
Imagine frequent fire alarms
Imagine frequent monitoring alerts

14. 14:15 – All alerts are bad
Two possibilities:
a) something bad has happened or will happen
b) it’s a useless alert

16. Chapter 2: Afternoon
on how I understood
why bad alerts happen

17. 15:03 – A minute of reading
Afternoon /ɑːftəˈnuːn/ – noun
The time from noon or lunchtime to evening

20. currently i don’t see a disk space issue

21. Currently I don’t
see a road issue
...

22. Trash attracts trash
Alerts attract alerts

24. oh, those are just alerts about my systems
=
oh, that’s just my trash in the forest

25. 16:30 – Failures multiply
Disk full →
logrotate creates an incomplete file →
never rotates the source file again
https://bugzilla.redhat.com/show_bug.cgi?id=1374550

26. 19:00 – What causes alerts?
Technical challenges
Lack of time
Humans
●Attitude?
●Culture?
●Organisation?

27. Chapter 3: Evening
on how I tried to
battle the alerts (and lost)

28. 00:13 - Let’s change the tools
Let’s use new alerting solution, new...
If emails can be ignored, so can be other alerts.
It’s culture, not the tool.

29. 01:00 – What’s this thing?
I must understand what I am running -
and monitoring
I must know my monitoring solution
●So I don’t spend two days writing a script to
replicate a feature
●“Haha, a week of training”

30. 01:20 – Do we know our tools?
Next version of grep rumoured to include graphs
Monitoring solution matters, but it is my decision
...I’ll try to pick one

32. From: somebody@example.com
Subject: Re: High: oldappserver.example.com Server unreachable
Logrotate is configured but files are not removed.
I deleted them.
--
Somebody
Starting to see a pattern here

34. not delay
(unless in a speeding bus)
Fix
I like that

35. Fix it for everybody
don’t hack “your server” only

36. Email
is not
"the worst notification method”

37. 02:40 – Rock rock till you drop
I will tirelessly work on making alerts:
●Less frequent
●More useful and meaningful

38. 02:50 – What’s a useful alert?
“System x down” is mostly clear, but others not
I’ll make no typos in alert instructions
write stupid, think stupid

39. 03:10 – Auto-resolve that crap
I estimate this will happen 3 more times
I add automatic action
Because I know it will happen 30 times

40. 03:30 – Monitoring is not a tasklist
These occurrences are useful to know about
So I’ll make it logged
If it really is useful, I can look at the log *
* usually nobody does

41. 03:50 – What’s that noise?
Do we really have
a problem with alerts?

42. 05:30 – We so trendy
Agile = “I won’t have to maintain it” *
Devops = “production is free-for-all” **
●Bring developers in on my monitoring
* Yes, it’s all misappropriated and misunderstood
** Still the same

43. Problems in /etc/motd * = “in your face”
* Credit for the idea to Ilya Ableev (Badoo)

44. 05:52 – Mind the language
False -
from Latin falsus (“counterfeit, false; falsehood”)
Never say “false positive”! (Thank you, Aaron)
Misconfiguration != false positive

46. "oh, it's a false positive" = oh, all is good here
"oh, it's misconfiguration" = how do we fix it?

47. 06:14 – Minding the language
Is "this is bad" fine?
In many countries, but maybe not in others
“Hate useless alerts” - Europe vs USA?

48. 06:20 – Tact filter?
Normal people with outgoing tact filter
Geeks with incoming tact filter
http://www.mit.edu/~jcb/tact.html

49. just a test

51. Most teams slip like cows on ice
if "cleanliness" rules are not
constantly refreshed

52. Make Alerts Meaningful Again
MAMA

53. Respect your oncall
Respect yourself

54. When I get mad
And I get pissed
I grab my pen
And I write out a list

55. □Oncall is alerted during maintenance
□Old systems are not cleaned up from monitoring
□Alert comments/instructions not too useful
□Alert comments/instructions with typos
□“false positive” is thrown around frequently
□Many monitoring solutions in place
□Problems are delayed, not fixed
□Problems are fixed in parts of the environment,