How to run long running jobs with Kamal

Sometimes background jobs take a while to complete but you need to deploy new versions in the meantime, what now?

Table of contents

Long running jobs

Perhaps the most obvious answer to running very long jobs with Kamal is not to run them at all. Your systems will generally do better if won't have any long running jobs to schedule. To do that you'll need to split your job to small workable units that can work independently or user pausable job processors.

However, if that's not the option for now and you are thinking how to do this anyways, here are two options you can try.

When you deploy new version of an application, Kamal will deploy its newly built containers and replace the old version in a blue-green fashion. However, redeploying accessories aren't part of this ceremony and are never interrupted. So what you could technically do as the first option is to use a current version of the application image and use it to run the job processor for long running jobs queue.

Kamal can also deploy specific application roles, so the second option would be to pass the explicit list of roles except the one that would run your long running jobs:

$ kamal deploy --roles web,fast_jobs

It's not perfect as you might lose track of what's going on. There are simply workarounds if you have some specific long running jobs lined up.

Author
Josef Strzibny
I am a full stack web developer with love for Linux. I made and deployed my first commercial web applications during high school in 2008. I have worked for Red Hat on the platform and developer experience teams as a Linux packager.

© Deploy Linux Blog