Presentation given at the "rails underground" conference about distributed and concurrent programming with ruby using RabbitMQ and EventMachine

1. “Divide and conquer
riding rabbits and
trading gems”
- a tale about distributed programming -
• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbit_starter
Paolo Negri @hungryblank

2. Resources
• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbit_starter

3. About me
Time
GNU/Linux - Dbs
Perl
“systems”
PHP
Python
Ruby

4. Summary:

5. Distributed
Concurrent
Programming

6. rabbitMQ
http://www.flickr.com/photos/myxi/448253580

7. Control

8. The problem
Given a dictionary of
1.000.000 search phrases,
compare for each one
the first page of results on
google.com and bing.com

9. Don’t do it.
It’s a violation of terms
of service!
(Sorry, I needed an example that
required little or no explanation)

10. Comparison

11. The whole process

12. Great! but...
• Fetching, parsing and analyzing
2.000.000 pages of results will take a
long time
• we want to collect the statistics in a
limited time and this sequential
approach is not quick enough

13. Distributed Computing
Is a method of solving computational
problem by dividing the problem into
many tasks run simultaneously on
many hardware or software systems
(Wikipedia)

14. Map Reduce "Map" step
the master node takes the
input, chops it up into smaller
sub-problems, and distributes
those to worker nodes.
(Wikipedia)

15. Problems:
• How many nodes?
• How many workers?
• Distribution mechanism to
feed the workers?

16. What about queuing?
• the master node takes the input, chops
it up into smaller sub-problems, and
publishes them in a queue
• workers independently consume the
content of the queue

17. Here comes
• RabbitMQ is an implementation of AMQP,
the emerging standard for high
performance enterprise messaging
• It’s opensource
• Written in Erlang

18. Erlang?
• general-purpose concurrent programming
language designed by Ericsson
• first version written by J. Armstrong in 1986
• distributed
• fault tolerant
• soft real time
• high availability

19. Erlang - is coming back
• Projects
• CouchDB - RESTful document storage
• WebMachine - REST toolkit
• Nitrogen - web framework
• Mochiweb - web framework
• Ruby shops using it
• 37 Signal - Campfire
• Heroku

20. + Erlang
It’s messages all the way down

21. Install it
• sudo apt-get install rabbitmq
• sudo gem install tmm1-amqp
Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4
to follow the code in the slides

22. Do it! - master node

23. Do it! - worker node

24. Get for free
• Decoupling master/worker
• Workers take care of feeding themselves
• Flexible number of workers

25. Behind the scenes
Worker1
Master
msg A Exchange Queue Worker2
Worker3

26. Behind the scenes
Worker1
Master
Exchange Queue Worker2
msg A
Worker3

27. Behind the scenes
Worker1
Master
Exchange Queue msg A Worker2
Worker3

28. Behind the scenes
msg A Worker1
Master
Exchange Queue Worker2
Worker3

29. RabbitMQ
• Multiple exchanges (and multiple types of
exchange)
• Multiple queues
• Queues are connected by bindings to
exchanges
• Exchanges route messages to queues

30. RabbitMQ
• Exchanges and queues have names
• BUT direct exchanges created implicitly are
not public and don’t have name
• Queues and messages are resident in RAM
and can be persisted on disk (at a
performance price)

31. What and where
Master
(ruby)
RabbitMQ
(Erlang) Worker
TCP/IP
(ruby)
Worker
(ruby)
Exchange Queue Worker
(ruby)

32. Problem #1
If a worker has a problem we might lose one
or more messages
http://www.flickr.com/photos/danzen/2288625136/

33. Solution - ACK in worker

34. Acknoledgement
• messages - 1 or more depending by
prefetch settings - are passed to the client
• the client processes and acknowledges the
messages one by one
• if the client connection closes =>
unacknowledged messages go back in the
queue

35. http://www.flickr.com/photos/streetfly_jz/2770586821/
Problem #2
No convenient way to control the workers

36. System queue - worker

37. System queue - control
• save it as system_command.rb
• ruby system_command.rb halt

38. System queue
Queue1 Worker1
Control script
msg A Exchange Queue2 Worker2
Queue3 Worker3

39. System queue
Queue1 Worker1
Control script
Exchange Queue2 Worker2
msg A
Queue3 Worker3

40. System queue
msg A Queue1 Worker1
Control script
Exchange msg A Queue2 Worker2
msg A Queue3 Worker3

41. EventMachine

42. EventMachine
Is an implementation of Reactor Pattern
• Non blocking IO and lightweight
concurrency
• eliminate the complexities of high-
performance threaded network
programming

43. without EM with EM
Ruby process Ruby process
Time
code code
network network
operation Free
operation
use network use network
operation operation
Callback
result result
Free
Free

44. EventMachine
amqp gem is built on top of
EventMachine => you’re in a
context where you can
leverage concurrent
programming

45. EM - Deferrables
“The Deferrable pattern allows you to specify
any number of Ruby code blocks that will be
executed at some future time when the status
of the Deferrable object changes “

46. EM - Deferrables

47. EM - Deferrables

48. EM - Deferrables

49. EM - Deferrables

50. Deferrables
without deferrables with deferrables
GooglePage GooglePage
BingPage
BingPage
Time

51. http://www.flickr.com/photos/philocrites/341850461/
Problem #3
How many of them?
what are they doing?

52. Heartbeat - worker

53. Heartbeat monitor

54. Heartbeat queue
Worker1 msg A
Worker2 Exchange Queue Monitor
Worker3
msg B

55. Heartbeat queue
Worker1
msg A
Worker2 Exchange Queue Monitor
msg B
Worker3

56. Heartbeat queue
Worker1
Worker2 Exchange Queue msg B msg A Monitor
Worker3

57. Clustering
RabbitMQ - node A RabbitMQ - node B
TCP/IP TCP/IP
RabbitMQ - node C
TCP/IP

58. async vs sync
Syncronous clients on github
• famoseagle/carrot
• celldee/bunny

59. Rabbits on github
• danielsdeleo/moqueue Tests/stubbing
• Check out ezmobius/nanite
“self assembling fabric of ruby daemons”
• Keep an eye on tonyg/rabbithub
implementation of pubsubhubbub (PubSub
over REST)
• auser/alice web app to monitor RabbitMQ

60. More rabbits on github
• tmm1/em-spec
• eventmachine/eventmachine
• tmm1/amqp
• macournoyer/thin

61. Q&A
?

62. Thanks!
(Thanks Mark!)
Paolo Negri
/ hungryblank