1.
Hi, I am Michi!
Head of Code at
@michilehr

2.
YOU CAN’T TEST EVERYTHING
BUT YOU SHOULD MONITOR IT!
11. October 2022 @ Nerd-BBQ

3.
"As Europe's leading photo booth provider, we have
made it our mission to help our brides and grooms
with their complete journey to their dream wedding.
This is something we work tirelessly on with our
team."
What is doing?
Philipp Schreiber - Co-Founder, KRUU.com

4.
Photo Booth
Cycle

5.
The Incident

6.
Good
~12 MB/s

7.
0.95 MB/s
Bad

8.
1. When did it start?
2. Why did it happen?
3. How to prevent?
4. How to notice early?
Investigate

9.
When did it start?
We had data in our Slack Channel, but…

10.
1. Write a script to extract the data as CSV
2. Import data to MySQL
3. Write query to aggregate by day
4. Create nice graph

11.
Started long time ago…

12.
What happened?
Network configuration error

13.
How to prevent?
No idea. Things like this happen

14.
How to notice early?

15.
How to notice early?

16.
Alerts!

17.
Query

18.
Trigger

19.
Notification

20.
● 404 alert by threshold
● Auth failure alert by threshold to
detect brute force
● …
What next?

21.
Thank you for your time!
Questions?
Feedback?
Notes?