RailsConf 2008 presentation: Asynchronous Processing with Ruby on Rails

1. Asynchronous Processing
Jonathan Dahl
(and RailSpikes,
Slantwise,
Zencoder,
etc.)

2. I.
What is it, and
why should I care?

4. Wife: What are you talking about at
RailsConf this year?
Jon: Asynchronous Processing
Wife: [changes subject]

5. important
tool

6. (ahem...)

8. Related Concepts
• Background Processing
• Parallel Processing
• Distributed Processing

9. has_attachment :storage => :s3

10. Image Upload ( 15 seconds)
Send to S3
Browser
Response

11. Image Upload
Send to S3 ( 15 seconds)
Browser
Response

12. Image Upload
Send to S3
Browser
( 3 seconds)
Response

13. Image Upload
Send to S3
Browser
Response

14. has_attachment :storage => :file_system

15. Image Upload ( 15 seconds)
Browser
Response
Send to S3

16. Image Upload
Browser
( 3 seconds)
Response
Send to S3 ( who cares?)

17. has_attachment :storage => :s3,
:thumbnails => {
:thumb => '100x100!',
:small => '240x180>',
:medium => '500x500>' }

18. Image Upload
Generate 3 Thumbnails
Send Thumbnail A to S3
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
Browser Response

19. Image Upload
Generate 3 Thumbnails
Send Thumbnail A to S3
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
Browser Response

20. Image Upload
Generate 3 Thumbnails
Send Thumbnail A to S3
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
Browser Response

21. Image Upload
Generate 3 Thumbnails
Send Thumbnail A to S3
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
Browser Response

22. Image Upload
Generate 3 Thumbnails
Send Thumbnail A to S3
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
Browser Response

23. Image Upload
Generate 3 Thumbnails
Send Thumbnail A to S3
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
Browser Response

24. Image Upload
Generate 3 Thumbnails
Send Thumbnail A to S3
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
Browser Response

26. II.
When do I need it?

27. Time

29. Request
• Method (GET, POST)
• URI (host, port, path)
• Parameters

30. Response
• Status (200, 404, 500)
• Metadata (content type, server info,
etc.)
• Body (xml, html, file)

31. Resources

32. Trigger

33. HTTP trigger - browser request

34. HTTP trigger - API request
GET /photos/1.xml HTTP/1.1
Host: example.com:80

35. Human trigger - capistrano
cap staging deploy

36. Human trigger - rake
rake db:migrate

37. Human trigger - console
$ script/console production
Loading production environment
(Rails 2.0.2)
>> Photo.destroy_all

38. No trigger?
- Send email in 2 hours
- Sync data at 3:00am PST
- Notify admin when disk is 90% full
- Expire sessions that are inactive
- Archive records that exceed quota

39. 1. Time
2. Resources
3. Trigger

40. Concrete examples
• Sending mail
• Transcoding video/audio
• Storing images on S3
• Receiving email
• Synching with outside database
• Complex computations

42. class Emailer < ActionMailer::ARMailer

45. Zencoder
User
Video Sharing
Website
Data Storage
(Amazon S3)
Zencoder
Manager
Worker Worker
Worker Worker

47. class Photo < ActiveRecord::Base
after_create :background_s3_upload
def background_s3_upload
Bj.submit quot;./script/runner ./jobs/send_to_s3.rb #{self.id}quot;
end
end

51. III.
So how do you
decide what to use?

52. be seamless

53. how reliable?

54. when should it run?

55. dependencies and
system complexity

56. scaling
and/or
performance

57. IV.
The simple solution:
fork or thread

61. Parallel vs. Background

62. 1. Stay within one
request

63. 2. thread.join

64. 3. ActiveRecord
ActiveRecord::Base.allow_concurrency = true

66. fire and forget

67. Spawn
spawn(:method => :fork) do
# do something
end

68. 1. Time
2. Resources
3. Trigger

70. V.
More robust
solutions

71. Task Storage
Task Trigger

72. Task Storage
• task details (what happens?)
• priority
• when to run

73. Task Trigger
• worker pulling jobs
• time-based
• execute immediately

74. Task Storage Task Trigger
Database daemon
Message Queue cron

75. Task Storage Task Trigger
Database daemon
Message Queue cron

76. create_table quot;jobsquot; do |t|
t.text quot;commandquot;
t.integer quot;priorityquot;
t.integer quot;pidquot;
t.datetime quot;submitted_atquot;
t.datetime quot;started_atquot;
t.datetime quot;finished_atquot;
t.text quot;resultquot;
end

77. create_table quot;photosquot; do |t|
t.string quot;filenamequot;
t.datetime quot;created_atquot;
t.datetime quot;processed_atquot;
end

78. create_table quot;photosquot; do |t|
t.string quot;filenamequot;
t.datetime quot;created_atquot;
t.datetime quot;processed_atquot;
end

79. Task Storage Task Trigger
Database daemon
Message Queue cron

80. Task Storage Task Trigger
Database daemon
Message Queue cron

81. • Amazon SQS
• Websphere MQ
• Starling
• JMS
• beanstalkd

82. queue = SQS.get_queue(quot;task_listquot;)

83. put message
queue.send_message quot;process:2872quot;

84. receive message
message = queue.receive_message

85. Starling
starling -h 192.168.1.1 -d

86. require 'memcache'
starling = MemCache.new('192.168.1.1:22122')
# Put messages onto a queue:
starling.set('my_queue', 12345)
# Get message from the queue:
starling.get('my_queue')

87. Task Storage Task Trigger
Database daemon
Message Queue cron

88. storage choice?
• queue: optimized for performance
• database: you’ve already got one

89. Task Storage Task Trigger
Database daemon
Message Queue cron

90. daemon
#!/usr/bin/env ruby
class JobRequester < SimpleDaemon::Base
def self.start
loop { Job.process_next }
end
end
JobRequester.daemonize

92. Task Storage Task Trigger
Database daemon
Message Queue cron

93. 0 6 * * * script/runner jobs/send_emails.rb

94. cronedit
require 'cronedit'
CronEdit::Crontab.Add quot;send-emailsquot;, {
:minute => 0,
:hour => 6,
:command => quot;script/runner jobs/send_emails.rbquot;
}
CronEdit::Crontab.Remove 'old-task'

95. trigger choice
• process: always running
• cron: as reliable as your operating
system

97. BackgroundDRb
class BillingWorker < BackgrounDRb::MetaWorker
set_worker_name :billing_worker
def create(args = nil)
# this method is called when worker is loaded for the first time
end
def charge_customer(customer_id = nil)
logger.info 'charging customer now'
end
end
MiddleMan.worker(:billing_worker).charge_customer(current_customer.id)

98. :backgroundrb:
:ip: 0.0.0.0
:development:
:backgroundrb:
:port: 11111
:log: foreground
:production:
:backgroundrb:
:port: 22222
:lazy_load: true
:debug_log: false
./script/backgroundrb start

99. AP4R
def MyController
def queue
ap4r.async_to({:action => 'download'}, {:story =>
story.id, :url => params[:url]})
end
def download
# long-running task
end
end

100. Bj
Acronym

101. create_table quot;bj_jobquot;, :primary_key => quot;bj
t.text quot;commandquot;
t.text quot;statequot;
t.integer quot;priorityquot;
t.text quot;tagquot;
t.integer quot;is_restartablequot;
t.text quot;submitterquot;
t.text quot;runnerquot;
t.integer quot;pidquot;
t.datetime quot;submitted_atquot;
t.datetime quot;started_atquot;
t.datetime quot;finished_atquot;
t.text quot;envquot;

102. Bj.submit quot;./script/runner ./jobs/task.rbquot;

103. after_create :bj_send_to_s3
def bj_send_to_s3
Bj.submit quot;./script/runner ./jobs/send.rb #{id}quot;
end

105. Workling
# environment config
Workling::Remote.dispatcher =
Workling::Remote::Runners::StarlingRunner.new

106. # task class
class ImageWorker < Workling::Base
def send_to_s3(options = {})
# put file to S3
end
end

107. # trigger asynchronous job
ImageWorker.asynch_send_to_s3(:image_id => 2927)

108. script/workling_starling_client start

109. Pitfalls

110. race conditions

111. alive, but stalled

112. VI.
some
recommendations

113. general purpose
Bj

114. distributed processing
SQS
(+ custom worker)

115. time-scheduled
cron
(+ rake or script)

116. speed + scalability
Starling/Workling

117. Thanks!
Jonathan Dahl
Slides at RailSpikes
http://railspikes.com