I often see applications implemented with separate task queues (e.g. Ruby's resque, the many MQ's, etc), but all background jobs I've written so far simply used the state of the database to figure out what to do next (e.g. a "last updated" timestamp can be used to determine which items to update next, setting a cache column to NULL to indicate it should be recalculated, etc).

I'm curious if there's any literature comparing these approaches.

@ayo the thing is, oftentimes your data source isn't a database

@newt I'd stuff it in the database because working with multiple data sources is a pain. :geblobcatshrug:

But granted, that's a use case where a separate task queue makes sense. I still often see otherwise single-DB applications using separate queues, though.

@ayo I've used databases as a queue as well. Two edge cases I found:

1. You might not need to actually persist the data

2. You want to schedule concurrent workers and not have overlap in tasks. This is functionally a job scheduler, rather than implement my own, I simply used celery with redis and called it a day.
Sign in to participate in the conversation

A lonely little town in the wider world of the fediverse.