I’m recently asked what my recent technology excitement is, of course in the context of programming. The name of Heroku came into my mind instantly. I’ve been using Heroku for all my Rails based applications recently. I’m amazed how easy it is to deploy and scale (to a degree) applications on to Heroku.
Heroku’s default way of running background jobs is using the delayed_job Ruby gem, which essentially records background jobs into a database table. And on Heroku, you can start a worker to monitor and process the job queue. This involves a lot of database polling as you could imagine.
So, in one of the applications I’m working on, I tried something more elegant (in my opinion). Github uses a Ruby gem called Resque for their background jobs.
- Resque uses a Redis based backend to store queued jobs, so it's fast in job logging and job retrieving.
- Resque allows you to separate jobs into different queues, so everything becomes better categorised.
- Resque also provides a Sinatra based web interface to monitor all jobs including failures, so better monitoring and better debugging.
All good. I also opted to use a Resque plugin called resque-heroku-autoscaler. This allows me to dynamically scale the number of Heroku workers up and down base on the jobs recorded in the queue. Since Heroku charges on worker usage, so with this auto scaler, I only pay for what’s used.
As for the implementation, it’s scattered across many files. Here’s my implementation, which handles the delivery of welcome emails upon membership creation.