1. Monitoring with
Syslog and
EventMachine

2. "It’s up doesn’t mean it’s working"

3. Monitoring

4. dashboards/monitoring

5. sketching
dashboards/monitoring

6. building
sketching
dashboards/monitoring

7. outlook
building
sketching
dashboards/monitoring

8. Dashboards

13. Motivation

14. Moving Target

15. Failure is always
an option

17. “Since the last deploy,
the number of signup
errors has gone up by
300 %. We might have
broken something.”

19. Counting
Things

20. Concurrent
Users

21. Signups

22. Logins

23. Errors

24. Sketch it

25. Application Application Application
Server Server Server
Event Aggregation
Dashboard

26. Criteria

27. Simple

28. Polyglot

29. Fire And
Forget

30. Criteria
Simple/Lightweight
Polyglot
Fire and Forget

31. Background story

33. Not Supported
node.js NewRelic RPM
erlang NewRelic RPM

34. Load Balancer
haproxy
node.js node.js node.js

35. haproxy logs
haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http
px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0
"GET /image.iso HTTP/1.0"

36. send logs custom syslog
haproxy server
NewRelic Ruby
New Relic
Agent

37. haproxy2rpm
http://github.com/wooga/haproxy2rpm

39. Syslog

40. Syslog
• Standard logging solution for Unix/Linux
• Facility (daemon, cron, user, local0, etc.)
• Priority/Level (Alert, Critical, Error,
Warning, etc.)
• Client and server
• Since 1980

41. ... It also provides devices
which would otherwise be
unable to communicate a
means to notify administrators
of problems or performance.
http://en.wikipedia.org/wiki/Syslog

42. Syslog Format
Date Hostname Program : Message
Jan 1 12:12:12 10.245.3.99 foo[421] : this is a message

43. Syslog client
# man logger
logger -p local0.notice -t HOSTIDM My Message

44. UDP

45. Fire And
Forget

46. Simple/Lightweight
Polyglot
Fire and Forget

47. Simple/Lightweight ✔
Polyglot
Fire and Forget

48. Simple/Lightweight ✔
Polyglot ✔
Fire and Forget

49. Simple/Lightweight ✔
Polyglot ✔
Fire and Forget ✔

50. Build it

51. send message
Event Emitter UDP : 514
Event Aggregator
Dashboard

52. send message
Event Emitter UDP : 514
Syslog
forward
Dashboard Event Aggregator

53. Emitting Events
require 'syslog'
def log(message, level = :warning)
script_name = $0
syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS
Syslog.open($0, syslog_option) do |s|
s.send(level, message)
end
end

54. Server
• Listen on UDP
• receive and parse syslog messages
• update a counter
• push it out to dashboards

55. Server
class UdpServer
HOST = '127.0.0.1'
PORT = '3333'
def self.run
EM::open_datagram_socket(HOST,
PORT,
Handler)
end
end

56. Handler
require 'eventmachine'
class Handler < EM::Connection
def receive_data(data)
puts "Received event: #{data.inspect}"
puts SyslogParser.parse(data)
end
end

57. Syslog Parser
https://github.com/jordansissel/experiments/blob/master/
ruby/eventmachine-speed/basic.rb

58. Web App

59. Event Source

60. Event Source
Format
data: Hello dashboardnn
event: userlogonn
data: {"username": "John123"}nn
event: updaten
data: {"username": "John123", "foo": "bar"}nn

61. Sinatra
require 'sinatra'
get '/stream' do
content_type 'text/event-stream'
stream(:keep_open) do |out|
end
end

62. class WebApp < Sinatra::Base
def self.connections
@connections ||= []
end
get '/stream' do
content_type 'text/event-stream'
stream(:keep_open) do |out|
WebApp.connections << out
end
end
end

63. class WebApp < Sinatra::Base
# send the message to all clients
def self.send_event(event = {})
connections.each do |c|
if c.closed
connections.delete(c)
else
c << "event: #{event[:event]}n"
c << "data: #{event[:data]}nn"
end
end
end
# here go sinatra routes ...
end

64. # disable sinatra's autorun
disable :run
EM.run do
WebApp.run!
UdpServer.run
Signal.trap("INT") { EventMachine.stop }
Signal.trap("TERM") { EventMachine.stop }
end

65. View

66. EventSource Basics
// Open up an event source socket
var source = new EventSource('/stream');
source.addEventListener('message', function(e) {
console.log("message:" + e.data);
}, false);
source.addEventListener('open', function(e) {
console.log('event source opened:' + e);
}, false);
source.addEventListener('error', function(e) {
console.log('received an error:' + e);
}, false);

67. Custom Events
source.addEventListener('login', function(e)
{
console.log("login: " + e.data);
}, false);
source.addEventListener('loginError',
function(e) {
console.log("login error: " + e.data);
}, false);

68. Testing
require 'socket'
desc 'send test data'
task :send_test_data do
socket = UDPSocket.new
host = "127.0.0.1"
port = "3333"
events = ["message", "login", "loginError"]
while true do
random_index = random_index(0..2)
random_value = rand(0..1000)
message = "#{events[random_index]}:#{random_value}"
socket.send(message, 0, host, port
sleep(1)
end
end

69. https://github.com/
phuesler/yada

70. Missing features
• Persistency
• Querying
• More complex aggregations (avg, timers,
etc.)
• Fancy visualization

72. Existing Solutions
• https://github.com/etsy/statsd
• https://github.com/noahhl/batsd
• https://github.com/eric/metriks
• http://square.github.com/cube/

73. Graphite
http://graphite.wikidot.com/screen-shots

74. Librato Metrics
https://metrics.librato.com/

75. Monitor all the
things!!!!

76. wooga.com/jobs
WANTED
dead or alive
BACKEND DEVELOPER
REWARD
www.wooga.com/ jobs/

77. Further Reading
• http://codeascraft.etsy.com/2011/02/15/
measure-anything-measure-everything/
• http://code.flickr.com/blog/2008/10/27/
counting-timing
• https://speakerdeck.com/u/roidrage/p/
metrics-monitoring-logging

78. Slides
http://www.slideshare.net/wooga

79. Credits
Cliff (Flickr)
401K (Flickr)
Scott Kidder (Flickr)