Recently I have been migrating a Rails application to Heroku. The application is a few years old and very large. Deploying to Heroku was fairly straight forward but within a short time I realised that Heroku’s maximum 30 timeout was going to be a problem.
Heroku’s timeout is completely non-negotiable and probably rightly so. Any application which is taking more than 30 seconds to load has some serious problems. Unfortunately, my application has some serious problems. Heroku compounds these problems in one swift move by using the swap file. Once an application starts using the swap file, processing time skyrockets.
So, here are a few ways I have used to try and alleviate the problem:
- Move any long or processor intensive tasks to a worker dyno. Using the delayed_job gem, I was able to pass a number of intensive tasks over to a heroku worker instance. The worker instance then churns away in the background freeing up the web instance to continue serving the user. Also, by using the workless gem, I was able to switch of the worker dyno when it wasn’t required, saving those all important $$.
- Pre-empt the heroku timeout to rescue the application. If the heroku timeout limit or your web server timeout limit are reached, you loose the ability to manage the problem within you application. By using the rack-timeout gem, you can catch the exception by setting the rack timeout to a second less than the web server timeout. You can then manage the exception within the application by logging data and displaying useful information to the user.
- Size your instance correctly to stop switching to swap file. If you are using a web server like unicorn, you can set the number of concurrent processes for users accessing you application. But beware, each instance will increase the memory usage on your dyno. New relic will give you a good indication of the average memory usage of you application. If you set unicorn concurrency then you can work out your required instance size [average_memory * concurrency = instance_size]. Quick tip – derailed_bencmarks gem shows the memory usage of your gems. It might help reduce your application size.
- If you hit the swap file, restart the dyno. Using the unicorn-worker-killer gem, you can set the dyno to restart if the memory quota is exceeded. This may also allow you to retry the page using the exception catching explained in ‘Step 2’. By doing
redirect_to response.pathin your exception rescue, you are effectively retrying the page with the newly restarted dyno.
Using these methods, you should be able to significantly reduce the impact of application timeouts if not avoid them completely.