1.
@papa_fire
Troubleshooting with monitoring
Testing in production
DevOps monitoring
[something] testing [something]
monitoring [something] in
production
Leon Fayer

2.
❖ @papa_fire
❖ leon@omniti.com
❖ fayerplay.com
❖ slideshare.net/LeonFayer1
THAT’S ME
WHO AM I?
๏ engineer for 20+ years
๏ professional cynic
๏ @ OmniTI
๏ build and operate big systems
๏ we are hiring!
๏ omniti.com/is/hiring

3.
@papa_fire
I HATE TESTING

4.
@papa_fire
testing is required

5.
@papa_fire
testing is not enough

6.
@papa_fire
> unit testing
> functional testing
> resilience testing
> performance testing
> …

7.
@papa_fire
testing can give a false
sense of security

8.
@papa_fire
testing is deterministic

9.
@papa_fire
data problem

10.
@papa_fire
> quantity of data
> frequency of data
> quality of data

11.
@papa_fire
example
Wolfe+585

12.
@papa_fire
example
Hubert Blaine Wolfeschlegelsteinhausenbergerdorffwelchevoralternwaren-
gewissenhaftschaferswessenschafewarenwohlgepflegeundsorgfaltigkeitbe
schutzenvorangreifendurchihrraubgierigfeindewelchevoralternzwolfhundert
tausendjahresvorandieerscheinenvonderersteerdemenschderraumschiff
genachtmittungsteinundsiebeniridiumelektrischmotorsgebrauchlichtalsseinur
sprungvonkraftgestartseinlangefahrthinzwischensternartigraumaufdersuchen
nachbarschaftdersternwelchegehabtbewohnbarplanetenkreisedrehensichundwo
hinderneuerassevonverstandigmenschlichkeitkonntefortpflanzenundsicher
freuenanlebenslanglichfreudeundruhemitnichteinfurchtvorangreifenvor
andererintelligentgeschopfsvonhinzwischensternartigraum, Sr.

13.
@papa_fire
user problem

14.
@papa_fire
“Users (n) - distributed fault injection
test suite for production

15.
@papa_fire
example
Corrupted Blood bug

16.
@papa_fire
example

17.
@papa_fire
other factors

18.
@papa_fire
> lack of foresight
(Y2K bug)
> too many use-cases
(female Tauren bug)
> change to assumptions

19.
@papa_fire
testing is great for
“known knowns”

20.
@papa_fire
testing is ok for
“known unknowns”

21.
@papa_fire
testing is bad for
“unknown unknowns”

22.
@papa_fire
enter monitoring

23.
@papa_fire
why monitor?

24.
@papa_fire
because testing isn’t
enough

25.
@papa_fire
> software is never perfect
> systems are complex
> external dependency worry
> proactive is better than reactive
> …

26.
@papa_fire
because things change

27.
@papa_fire
because things change
in production

28.
@papa_fire
what to monitor?

29.
@papa_fire
in God we trust all others
we monitor
“

30.
@papa_fire
> systems
> databases
> applications
> integration points
> performance
> user behavior
> …

31.
@papa_fire
is it enough?

32.
@papa_fire
is it too much?

33.
@papa_fire
what is important?

34.
@papa_fire
what is important?
(i.e. what to alert on)

35.
@papa_fire
example
> servers up and running
> HTTP checks return 200
> tweets are lost

36.
@papa_fire
s/system checks/unit tests/

37.
@papa_fire
I don’t give a **** if the
datacenter is on fire as
long as I am still making
money
“ — CEO

38.
@papa_fire
we monitor because things
change

39.
@papa_fire
changes effect business

40.
@papa_fire
top-down approach
> understand business
> define baseline
> correlate data

41.
@papa_fire
example
๏ online marketing company
๏ major e-commerce component
๏ ~100 million users
๏ 1 billion emails/month
๏ 300,000 lines of code
๏5600+ metrics collected

42.
@papa_fire
it all starts with a call …

43.
@papa_fire
revenue

44.
@papa_fire
revenue + traffic

45.
@papa_fire
revenue + traffic + load time

46.
@papa_fire
revenue + traffic + load time + db

47.
@papa_fire
revenue + traffic + load time + db + email

48.
@papa_fire
… email wasn’t monitored?
what if …

49.
@papa_fire
… email wasn’t monitored?
(it would be after this)
what if …

50.
@papa_fire
instrumentation
is never done

51.
@papa_fire
example
> same symptoms
> higher decline rates
> all metrics are within norm

52.
@papa_fire
example
> same symptoms
> higher decline rates
> all metrics are within norm
AmEx blocked

53.
@papa_fire
tl;dr

54.
@papa_fire
testing and monitoring
not
testing or monitoring

55.
@papa_fire
understand the business

56.
@papa_fire
continuous improvement

57.
@papa_fire
{also bad at conclusions}

58.
@papa_fire
THANK YOU
questions?