Redis job queues for work that shouldn’t make users wait

Sending a batch of reminder emails or waiting on a slow API shouldn’t freeze the screen. Without a queue, a failed email send is lost when the request ends.

Reviewed by: Rahul Jaiswar, founder

How the queue pattern works

The request saves what the user asked for, adds a job to a queue and responds straight away. A worker picks the job up, does the slow part, and retries with a growing delay if it fails. The screen checks back for the result or gets notified. This is how long work stays out of page requests, API calls and serverless functions, whichever framework the app uses.

Redis and BullMQ in brief

Redis is an in-memory data store used for caching, counters and queues. BullMQ is a Node.js library that builds job queues on Redis, with retries, delays and scheduled jobs.

Where we use it

Microsive ATS runs its job queues on Redis with BullMQ.

What we check

We test two people saving the same record at once, not only one action after another. A lost update leaves no error behind, so a sequential test will not show it.

Do you need a queue at all?

A handful of background tasks a day can run from a database job table or a scheduled task, with no extra service to run. And if a page is slow because a query lacks an index, a cache in front only hides it; fix the query first. Keep important records in PostgreSQL.

Settings and habits that lose jobs

Eviction deletes jobs. BullMQ expects Redis’s maxmemory-policy to be noeviction. With an evicting policy, Redis can drop queue keys under memory pressure with no error in the app. Check your managed provider’s default eviction policy before the first job runs.
Jobs that run twice. After a crash or a stalled lock, a job can run again. Write jobs so a repeat has no side effect, such as a second email.
Redis on a public port. Redis belongs on a private network with a password.

To commission an app with background jobs, see web apps.

Which actions make your users wait?

Tell us what happens behind them and how long it takes today.

Tell us what’s slow