Heroku and jobs on a timer...

So in the intervening time since my last post, I started a new project in Ruby on Rails and I have to say I am enjoying the framework . Getting used to Ruby wasn't as hard as I thought although I am sure I am writing Ruby too much like Python :)

One of the great things about using Ruby is the plethora of hosting options as compared to Python Turbogears (where my own server was pretty much the only option). For the current project I am working I chose to take a look at hosting with Heroku which is what I guess I'd call a hybrid PAAS (Platform As A Service). Hybrid b/c it lets you use a database unlike Google App Engine, but it does not give you root access to a machine. Basically Heroku expects you to be using rails, and lets you do all the "railsy" things you would expect like running rake tasks.


Heroku has been great, though there are two things I'll point out for those intersted

1. Heroku only supports Postgres, so start your project with it! I didn't, but I also wasn't too far along so was able to port my data over from Mysql fairly easily.
2. If you are running Windows, be prepared for a bit of fight.

(2) may be more of a Rails thing than Heroku, but a LOT of the Rails plugins expect you to be running *nix Point in fact a lot of them expect a full *nix build environment. So, my advice, make sure you are running Cygwin, and specifically make sure you pull down basically all the DB libraries (to build both the Postgres and the Mysql adapters I needed them, and to build Taps, a DB migration library Heroku uses, you'll need Sqllite3).

Nugget: Once you have all these libs, you may get the following error from Cygwin: Unable to remap [libary name], do the following

1.) Shutdown all bash shells and X windows
2.) In command prompt run "ash rebaseall"
3.) Wait to finish
4.) Profit (I mean you should be good to go)

So once I finally got all that Windows specific stuff put to bed I got my application up and running on Heroku, but I needed something that would schedule and run a task every minute. Heroku offers a Cron Job API, but the most often it will run is every hour, which is not sufficient for my needs. However Heroku does offer the DelayedJob plugin for Rails, could I somehow use that to accomplish the same task?

Short answer is yes. DelayedJob offers an input parameter called "run_at" which allows you to not only delay the job but schedule when it should run. By some clever utilization of this we are able to run a job every minute. How? Here's the code.

First off we need to add the first delayed job when the application starts up, so in environment.rb add the following right before the end of the "Rails::Initializer.run do |config|" block.

config.after_initialize do
Delayed::Job.all.each do |old_job|
 old_job.destroy
end
Delayed::Job.enqueue SocialChecker.new(nil), 3
end

This will delete any old jobs that are hanging around in the database (this is important, you'll see why later) and queues up a new job to be run immediately (I didn't set the run_at time for this job).

Now in the "perform" method of my SocialChecker class (that is how DelayedJob works, you must have a perform method in the class you want to enqueue, and that method is called when the job is run) I have the following.

def perform
log_info("Checking: %s" % Time.now)
check()
new_time = Time.now.advance(:minutes => 1)
Delayed::Job.enqueue SocialChecker.new(nil), 3, new_time
end

This little block performs the "check()" method, which is the meat of the things I actually want to do on a one minute timer. The important part is after the "check()" method, here I create a Time object and make it one minute later than the current time. Then I queue up a new job setting the run_at to the one-minute-later Time object. So what this does is it always adds a new job to the DelayedJob queue and that new job is always one minute in the future, so when the "new" job gets executed, it'll add yet another new job to the queue one minute in the future and on and on.... The overall effect is of a scheduled task running every minute (in my case, but just change "Time.now.advance(:minutes => 1)" to whatever you need.

So lets talk potential problems. First off I think one can see why I added the code to delete leftover jobs on initialize, because if there are leftover jobs, when they run each of them will add a new Job for a minute later so we could get into a situation where we would get a Job explosion. The deletion code just keeps us clean on startup. To be safe one could add that code block into the perform method as well, to also make sure we are clean before we add another job.

One other problem is the potential failure of the perform task (assuming the error is not caught, which of course it should be), one might think this would mean that our future jobs wouldn't get pushed into the queue and we'd lose our scheduled task. That is partially right, if the perform task keeps failing then we would not push a new Job into the queue, but one of the great things about DelayedJob is that it will retry your Jobs with an exponential backdown. So as long as your job doesn't continuously fail (which of course is a much bigger issue), the schedule task should recover, eventually.

Note that to use background tasks in Heroku one must pay $15/month and Heroku will give you "1 thread" to consume your background jobs. So if you have a lot of scheduled jobs in mind then they might conflict this might not work that well. My hope would be that eventually Heroku will allow us to purchase more threads to consume the jobs, allowing this process to scale.

So one thing I am bit worried about is, because of their interesting mechanisms for sharing load among servers, whether Heroku wants people doing these short time-scale Jobs. I am heartened by their documentations saying that one gets a "thread" to perform your Jobs, so my assumption is once they give you a thread it'll hang around and you may as well use it. Hopefully someone from Heroku will get back to me if there is a huge problem with this, it is not my intention to bypass their restrictions, at least not if they are there for a good reason ;)

Hope this helps people looking for some Heroku info, please leave some comments or twitter me @jostheim to help improve my code above.