Learn the theory
Job Queues with BullMQ (Redis): how it works
4 min read · Updated
Some work shouldn't happen inside a web request: sending email, resizing images, calling slow APIs, generating reports. A job queue lets the request hand the work off and return immediately while workers process it in the background. BullMQ is the popular Node.js queue built on Redis: jobs and their states live in Redis data structures, workers are stateless processes that take jobs, and the library handles retries, delays, priorities, rate limits and crashed workers for you.
In short
- A job moves through waiting → active → completed/failed, with delayed for scheduled jobs and retry backoff.
- All state lives in Redis, so jobs survive deploys and crashes; workers can be scaled horizontally.
- Retries with exponential backoff handle transient errors; a failed set holds jobs that ran out of attempts.
- If a worker dies mid-job, its lock expires and the job is detected as stalled and re-queued — so handlers must be idempotent.
- Use concurrency, priority and rate limiters to control throughput and protect downstream services.
Why queues?
- Fast responses: accept the request, enqueue, reply in milliseconds.
- Reliability: work survives restarts and is retried automatically.
- Smoothing bursts: a spike becomes a queue depth instead of an outage.
- Scale: add workers without changing the producers.
import { Queue, Worker } from 'bullmq';
const emails = new Queue('emails', { connection });
await emails.add('welcome', { userId: 42 }, { attempts: 4, backoff: { type: 'exponential', delay: 2000 } });
new Worker('emails', async (job) => sendEmail(job.data), { connection, concurrency: 5 });The job lifecycle
| State | Meaning | Redis structure (BullMQ) |
|---|---|---|
| waiting | Ready to be picked up (ordered by priority, then FIFO). | bull:<queue>:wait (list) / prioritized (zset) |
| delayed | Waiting for a time: an explicit delay or a retry backoff. | bull:<queue>:delayed (sorted set by due time) |
| active | Locked by a worker that is running it. | bull:<queue>:active (list) + a lock key with a TTL |
| completed | Finished successfully (kept for a while for inspection). | bull:<queue>:completed (sorted set) |
| failed | Out of attempts (kept with the error and stack trace). | bull:<queue>:failed (sorted set) |
Each job's data and options are stored in a Redis hash (bull:<queue>:<id>). Workers use atomic Lua scripts to move jobs between states, which is why two workers never process the same job.
Retries, backoff and poison jobs
When a job throws, BullMQ increments its attempt counter. If attempts remain it moves the job to delayed for the backoff period, then back to waiting.
- Fixed backoff: the same delay every time.
- Exponential backoff:
delay × 2^(attempt-1), e.g. 2 s, 4 s, 8 s — gives a struggling dependency time to recover and avoids hammering it. - Poison jobs can never succeed (bad payload, bug). After the last attempt they land in the failed set. Alert on that set and don't retry forever.
Crashed workers and stalled jobs
While a worker runs a job it keeps renewing a lock in Redis. If the process crashes or hangs, the lock stops being renewed and expires. A stalled-job checker notices an active job with no lock and moves it back to waiting so another worker can take it.
The consequence: a job can run more than once (the first attempt may have done part of the work). Make job handlers idempotent — the same advice as for webhooks.
Concurrency, priority and rate limiting
- Concurrency: how many jobs one worker runs at once (I/O-bound work benefits from higher numbers). Add more worker processes to scale further.
- Priority: lower number = served first, so a password-reset email overtakes a newsletter backlog.
- Rate limiter: cap jobs started per time window to respect third-party API limits.
- Delayed jobs:
delayschedules work for later, e.g. “remind the user in 3 days”. - Repeatable jobs: cron-like schedules stored in Redis, shared by all workers.
Running queues in production
- Use a Redis with persistence (AOF) and
maxmemory-policy noevictionfor queue data — evicting job keys loses work. - Monitor queue depth, job age, failure rate and worker count (Bull Board or Taskforce UI).
- Set sensible job retention (
removeOnComplete,removeOnFail) to bound memory. - Handle graceful shutdown: let the worker finish active jobs before exiting.
- Keep payloads small — store IDs, not whole objects.
Frequently asked questions
- What is BullMQ?
- A Node.js library for building job queues on top of Redis. It stores jobs and their states in Redis and provides workers, retries, delays, priorities, rate limiting and repeatable jobs.
- What are the job states in BullMQ?
- Waiting, delayed, active, completed and failed (plus waiting-children and prioritized for advanced flows). Jobs move between them via atomic Redis scripts.
- How do retries and backoff work in BullMQ?
- Set
attemptsand abackoffoption on the job. When it fails and attempts remain, it goes to the delayed set for the backoff time (fixed or exponential), then back to waiting. - What is a stalled job?
- A job that is marked active but whose worker stopped renewing its lock, usually because the process crashed or the event loop was blocked. BullMQ re-queues it, so it may run more than once.
- How do I scale BullMQ workers?
- Increase the
concurrencyoption for I/O-bound work and run more worker processes or containers. They all pull from the same Redis queue without coordinating. - Should I use BullMQ or RabbitMQ?
- BullMQ is the simple choice if you already use Redis and Node.js and want job semantics (retries, delays, priorities). RabbitMQ is a general-purpose broker with rich routing and works across languages; it suits complex messaging topologies.