Achieving Common Ground via Logging Slides from - http://devopsdays.org/events/2013-newyork/proposals/Achieving%20common%20ground%20via%20Logging/

1. Achieving
Common Ground via
Logging

2. Second Class Citizens

3. Why?

4. Why?
(?:(?:(?:b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|
Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)b)
+(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?:(?![0-9])(?:(?:2[0123]|[01]
[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])))
(?:(?:(?:b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]
{0,62}))*(.?|b))|(?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:
25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})
[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?(?(?:[w._/%-]+))(?:
[(?b(?:[1-9][0-9]*)b)])?): (?(?![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]
{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?
[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?(?:[+-]?(?:
[0-9]+))) [(?(?(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?b(?:Jan(?:uary)?|
Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|
Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)b)/(?[0-9]+):(?(?![0-9])
(?(?:2[0123]|[01][0-9])):(?(?:[0-5][0-9]))(?::(?(?:(?:[0-5][0-9]|60)(?:[.,]
[0-9]+)?)))(?![0-9])).(?(?:[+-]?(?:[0-9]+))))] (?S+) (?S+)/(?S+) (?(?:[+-]?(?:
[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?S+) (?S+) (?S+) (?(?:[+-]?(?:[0-9]+)))/(?(?:
[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+)))/(?S+) (?(?:[+-]?
(?:[0-9]+)))/(?(?:[+-]?(?:[0-9]+))))
Thank you @jordansissel

5. All good?
Yes. But...
Dev
Ops
Support
Products

6. Where to begin?
“Request Completed”
Masterzen - 10 Commandments of Logging

7. Context
Request (f67a) completed for user
George-Michael (token: weqeqweq)
to fakeblock in 3.16275 seconds.

8. Context
{
}
"msg": "Request completed",
"runtime_s": 3.16275,
"request": {
"action": "fakeblock",
"accessToken": "yjfhakfsf",
"user": "George-Michael"
},
"requestId": "f67a"

9. Context + Correlation
{
{
...
"host": "load-balancer",
"status": "200",
"request": "POST /url/ HTTP/1.1",
"http_user_agent": "iphone",
"x_requestId": "f67a",
"request_time": 3.9
....
"msg": "Request completed",
"runtime_s": 3.16275,
"request": {
"action": "faceblock",
"accessToken": "yjfhakfsf",
"user": "George-Michael"
},
"requestId": "f67a",
"host": "i-90zas2",
}
}

10. Ship them!

11. level : ‘error’
... and class : ‘mailer’
... and host : ‘i-8904’

12. action : ‘fakeblock’
... and runtime > 2.1

13. level : ‘error’
... and ...
... and ...
is count > threshold
Critical, Warn, OK?

14. Wait a sec...
{
}
"msg": "Request completed",
"runtime_s": 3.16275,
"request": {
"action": "fakeblock",
"accessToken": "yjfhakfsf",
"user": "George-Michael"
},
"requestId": "f67a"

15. type : ‘user_log’
and
user : ‘George-Michael’

16. “invalid identifier returned”

17. But we are a small
team...

18. Defaults work well
• 40 GB index/day
• 3k events/sec indexed
• m1.xlarge x 2
• Chef Cookbook available
• pyes (ES Api in py)

19. Bonus

20. Thanks
Manas Gupta
@ngrep_q