Okay, my talk is called “...” which I think is pretty descriptive. Except that it doesn’t really demonstrate the pure awesome is that is Resque, so really, we’ll call it...
Yeah.
So, why should you care about what I’m going to say in the next 20 minutes? Because when you walk away from this room, you’ll all be experts at building a network of high-performance and fault-tolerant workers.
So we’re going to talk about three things today
First, we’re going to do a general background of background processing
Next, we’ll cover the basic concepts of Resque that are derived from Delayed::Job
Last, we’ll get into Resque, and how it’s different.
So, why run background tasks at all?
- Resize
- Etc
That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- Resize
- Etc
That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- Resize
- Etc
That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- Resize
- Etc
That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
Let’s do something later than now. If my application doesn’t NEED to do this stuff right away, don’t keep the request tied up. After all, what IS a request for?
Webpages
In case you’d forgotten
- Rails serves webpages.
It’s not built to process images or send out mass e-mails; it’s built to do two things:
Webpages
In case you’d forgotten
- Rails serves webpages.
It’s not built to process images or send out mass e-mails; it’s built to do two things:
Webpages
In case you’d forgotten
- Rails serves webpages.
It’s not built to process images or send out mass e-mails; it’s built to do two things:
Webpages
In case you’d forgotten
- Rails serves webpages.
It’s not built to process images or send out mass e-mails; it’s built to do two things:
- Receive requests
- Send responses
- Receive requests
- Send responses
Everything else can go elsewhere
So... where should it go? There are a whole bunch of options.
- CRON
- Etc.
But sometimes those setups have problems
So... where should it go? There are a whole bunch of options.
- CRON
- Etc.
But sometimes those setups have problems
So... where should it go? There are a whole bunch of options.
- CRON
- Etc.
But sometimes those setups have problems
So... where should it go? There are a whole bunch of options.
- CRON
- Etc.
But sometimes those setups have problems
- They can fail without telling you why
- You can’t get the status of one of a worker
- Ruby processes can be unstable and crash-ey
So maybe you need something a little bit better
- They can fail without telling you why
- You can’t get the status of one of a worker
- Ruby processes can be unstable and crash-ey
So maybe you need something a little bit better
- They can fail without telling you why
- You can’t get the status of one of a worker
- Ruby processes can be unstable and crash-ey
So maybe you need something a little bit better
Delayed::Job to the rescue!
A Delayed::Job is an ActiveRecord model, so it lives in the database.
- You get persistence baked right in
- It’s unbelievably easy to use
A Delayed::Job is an ActiveRecord model, so it lives in the database.
- You get persistence baked right in
- It’s unbelievably easy to use
How easy?
Super easy. I pulled this straight from the github page. This code shows how you would create a Delayed::Job
But where does that job go?
Here - your database. Notice that it simply marshals the object and stuffs in the database.
So what do you do with these jobs in the database?
- Run the delayed_job script
This will create...
Workers. They’re separate Ruby processes that
- Live in their own thread
- Find jobs in the database
- Perform them separately from the request/response cycle
They boot up your Rails environment to do so.
Workers. They’re separate Ruby processes that
- Live in their own thread
- Find jobs in the database
- Perform them separately from the request/response cycle
They boot up your Rails environment to do so.
Workers. They’re separate Ruby processes that
- Live in their own thread
- Find jobs in the database
- Perform them separately from the request/response cycle
They boot up your Rails environment to do so.
So before, you had requests coming from a browser into Rails, which dispatches to various services, gets results, and sends something back to the browser
Now you have Rails storing the jobs for later, and Delayed::Job comes along and does the processing Rails isn’t good at doing anyway.
So Delayed::Job is totally rad. It handles most of what you might want it to handle.
So when we start in with Resque, you might wonder, “Isn’t Delayed::Job enough?”
Apparently not. There are some situations where Delayed::Job isn’t sufficient.
Basically, Resque is enterprise grade shit.
So the first big difference between Resque and Delayed::Job is that Resque doesn’t rely on ActiveRecord. It runs on something called Redis, which probably deserves
- its own tangent
Redis is an in-memory Key-Value Store similar to Memcached. But it offers
Redis is an in-memory Key-Value Store similar to Memcached. But it offers
Redis is an in-memory Key-Value Store similar to Memcached. But it offers
Redis is an in-memory Key-Value Store similar to Memcached. But it offers
Redis is an in-memory Key-Value Store similar to Memcached. But it offers
Redis is an in-memory Key-Value Store similar to Memcached. But it offers
Persistence! So you can reboot your server and Redis will reload your original queued jobs. It also is great at
Scalability! The Ruby gem will load-balance tasks across multiple Redis servers for you, meaning you spend less time configuring and more time queueing
So Redis makes Resque incredibly fast and easily distributed.
Also, using Redis means you don’t have to worry about ActiveRecord - so you can skip Rails entirely
The next big difference is Resque’s workers
- When a worker finds a job to process, it creates a fork that it can monitor to process the job
If a fork spirals out, it doesn’t take the worker with it
- Workers are capable of moving failed jobs into the failed List and moving on with their lives
- Workers are also capable of timing their own processes out and failing them
The next big difference is Resque’s workers
- When a worker finds a job to process, it creates a fork that it can monitor to process the job
If a fork spirals out, it doesn’t take the worker with it
- Workers are capable of moving failed jobs into the failed List and moving on with their lives
- Workers are also capable of timing their own processes out and failing them
The next big difference is Resque’s workers
- When a worker finds a job to process, it creates a fork that it can monitor to process the job
If a fork spirals out, it doesn’t take the worker with it
- Workers are capable of moving failed jobs into the failed List and moving on with their lives
- Workers are also capable of timing their own processes out and failing them
Resque also supports queues, so different types of tasks can be divided by disparate workers. A file server, for example, can have a worker that only zips files - while a web server can warm image caches.
Resque also skips the marshaling step, and instead opts for a class and a JSON-ifiable argument list. This will keep all instances current
The class’ “perform” method is called with the arguments you enqueued it with
Resque comes with its own interface for monitoring queues and tasks, and reviewing failed tasks.
So why use Resque? Use Resque if
- You have many, many, many background tasks
- You need to have different servers performing different types tasks
- You need bullet-proof workers
So why use Resque? Use Resque if
- You have many, many, many background tasks
- You need to have different servers performing different types tasks
- You need bullet-proof workers
So why use Resque? Use Resque if
- You have many, many, many background tasks
- You need to have different servers performing different types tasks
- You need bullet-proof workers
Trouble
- Complex / expensive
- No scheduling or priority
- Asynchronous Redis writes don’t guarantee persistence
Trouble
- Complex / expensive
- No scheduling or priority
- Asynchronous Redis writes don’t guarantee persistence
Trouble
- Complex / expensive
- No scheduling or priority
- Asynchronous Redis writes don’t guarantee persistence
Okay, if we have time I’ll show off Resque’s UI and do a quick comparison of it to D::J
Thanks for listening!
This is me - you can find me on Github or Twitter, and if you’re interested I’ll post these slides on Twitter.
Last but not least, I’m a visiting emissary from B’more on Rails, and I’d like to personally invite all of you to come up to our Meetups (on the 2nd tuesday of every month) and our Open Source Hack Nights (on the 4th Tuesday of every month).