Job within a Job: Scheduling Inception with Laravel Queues

This may seem obvious to many of you, but I always think it's worth talking about the obvious things in case it helps someone. I couldn't possibly count how many times I've been too focused on the wrong thing only to find a simpler solution has been staring me in the face the whole time.

Queues & Jobs

Laravel's Queues are extremely powerful. They're the perfect way to offload the heavier parts of your app's processing, speeding up your request-response times and making your app's user interface feel really snappy.

I don't want to encourage premature optimisation, but I've found a lot of the domain-specific parts of the apps I'm building make sense as discrete Jobs. So I'm beginning to come around to the idea of an almost Job-first style to writing my code.

It was only whilst I was building Ensemble that I realised just how powerful this approach is: These parts of my app can be run in whichever way I like and that I can change the behaviour with just a single line of code in some cases.

This is so freeing! I can easily switch from synchronous Jobs that run as a direct result of a user request, to an asynchronous Job that gets added to a queue.

For the code samples, let's imagine we're parsing XML feeds from different eCommerce stores to create a price comparison engine:

public function update($id)

{

  // Async

  ParseFeedJob::dispatch(Store::find($id))

      ->onQueue('feed-parse-workers');

  

  // Sync

  ParseFeedJob::dispatchNow(Store::find($id));

}

Now, regardless of how these Jobs get processed (sync or async), firing them only from a Controller method (although perfectly fine) limits the usefulness of writing Jobs - they're basically just extensions of the Controller, maybe ticking the 'thin controllers' box but betraying the spirit of that principle.

So, let's put our Jobs somewhere else...

The Task Scheduler

It's no coincidence that in the Laravel docs, Task Scheduling appears immediately after the docs on Queues. It's the next logical step really as chances are high that any Job is likely to be repeated on some sort of schedule.

The Scheduler makes this ridiculously simple:

$schedule->job(new ParseFeedJob(...))

  ->daily();

But perhaps you've already identified a problem? Our ParseFeedJob expects an instance of Store to be passed into it. The problem is, in the context of the Scheduler, we know nothing about an individual Store or the current user or a Request.

This is all because the Scheduler is run from cron and via an Artisan command: php artisan schedule:run and the console application kernel differs from the web application kernel - as it should, they're completely different paradigms.

So how do we overcome this 'problem'?

First, let's identify what we're really trying to do. Initially, we were running the ParseFeedJob from a controller. Let's assume that it was firing from a POST to some route like /stores/941/feeds/update or something.

So we were able to tell which Store it was for easily from the request. Now if we only had this one Store and we weren't worried about it being dynamic, we could do this and be done:

$schedule->job(new ParseFeedJob(Store::find(941)))

  ->daily();

But we do want it to be dynamic. We have many stores and firing off all of those POST requests manually every day (even if they were aggregated via another overall POST) is just super unnecessary.

So we want something that can parse all the Feeds of all the Stores. It sounds like that could be a Job all of its own! E.g. ParseAllFeeds

How about something like this?

class ParseAllFeedsJob implements ShouldQueue

{

  public function handle()

  {

    foreach (Stores::all() as $store) {

      ParseFeedJob::dispatch($store);

    }

  }

}

Well that was easy! We've now created a Job that creates other Jobs. Holy inception, Batman!

Dream within a dream

Hopefully you can see that this is useful. What I personally find especially great about this approach is that I can still use ParseFeedJob in controllers or other places (sync/async) when I know which Store I'm dealing with - for instance, if I want to fire the Job manually at some point in case it's failed a number of times and we're in between scheduled runs.

Also, this can work in any environment, whatever our Queue configuration looks like - either we have the capacity to push Jobs onto Queues handled by workers, or we don't and the Jobs have to run synchronously.

Even in dev environments where we don't have a cron set up to run the Task Scheduler, we can mimic its behaviour by simply firing the 'parent' Jobs via Tinker:

$ php artisan tinker

$ App\Jobs\ParseAllFeedsJob::dispatchNow()

It's super flexible, but it also helps us stay DRY and makes our code very self-descriptive.

Next time you're thinking about writing some heavy-weight code and worrying about where it should go, try putting it in a Job. And don't be afraid to create Jobs in Jobs!