Background Jobs and Queues for Reliable Web Apps

Sending an email, generating a report, or importing a large CSV should not keep an interactive request open unnecessarily. Background processing can shorten that request, but it introduces a new contract: the server accepted responsibility for work that has not finished. Reliability depends on how that responsibility survives crashes, retries, and duplicate delivery.

Define the job contract before adding a queue

Use the JSON to TypeScript tool to sketch types from an example job payload. Keep payloads small and explicit: a job identifier, tenant identifier, resource identifier, and schema version are often more useful than a copy of an entire database record. Generated types still need runtime validation at the queue boundary.

Distinguish the API request, the durable job record, and each execution attempt. A retry is usually another attempt at the same logical job, not a new business action. This distinction supports idempotency, auditing, and understandable progress reporting.

Write down when a job becomes accepted, how users check progress, and what success means. For an email, success may mean that a provider accepted delivery, not that a person opened the message. For an export, success should mean the completed artifact is available and access-controlled.

Avoid putting secrets or unnecessary personal data in messages. Queue dashboards, backups, and dead-letter storage can retain payloads longer than expected. Prefer references and fetch the authorized data inside the worker, while defining what should happen if that data changes or disappears before execution.

Accept work durably and expose its status

A queue-backed endpoint commonly returns 202 Accepted with a job identifier and a status URL. This indicates acceptance, not completion. Follow the semantics described in the MDN 202 reference, and return an error if you cannot durably record responsibility for the work.

Store job states such as queued, running, succeeded, and failed. Add timestamps and a safe error code, not a raw exception containing credentials. Clients can poll with backoff, while more demanding products can add notifications. Polling frequency should reflect expected duration rather than creating constant database pressure.

The dangerous gap is a database commit followed by a failed queue publish. Your order exists, but its receipt job never arrives. Reversing the order creates the opposite risk: a worker sees a message for an uncommitted or rolled-back record.

Use a transactional outbox when the business change and job creation must be atomic. Commit the business record and an outbox row in one database transaction. A relay publishes pending rows to the queue and marks them delivered. Because publishing and marking are separate operations, duplicates remain possible. The AWS transactional outbox guidance explains this failure boundary in detail.

Assume delivery can happen more than once

Most practical worker designs should tolerate at-least-once delivery. A worker may finish its side effect and crash before acknowledging the message. The queue then redelivers it. A long task may also outlive its visibility window or lease, allowing another worker to begin.

For effects contained within one database, use a unique business key and a transaction. The following SQL sketch makes granting credits and recording completion atomic; the application binds the placeholders and validates the job first:

BEGIN;
WITH claimed AS (
INSERT INTO processed_jobs (job_id)
VALUES ($1)
ON CONFLICT (job_id) DO NOTHING
RETURNING job_id
)
UPDATE accounts
SET credits = credits + $2
WHERE id = $3
AND EXISTS (SELECT 1 FROM claimed);
COMMIT;

This assumes a unique constraint on processed_jobs.job_id, a valid account, and a single transactional database. Production code must verify the target update and roll back if business invariants fail. Acknowledge the queue message only after commit. Retain deduplication records for the relevant replay horizon.

External effects require another strategy. Pass a stable idempotency key to a payment or messaging provider when supported. A local completion flag alone cannot close the gap between a remote success and a local crash. Do not advertise exactly-once effects unless every participating system actually supports that guarantee.

Bound retries, timeouts, and concurrency

Classify failures before retrying. Temporary network errors and rate limits may recover; malformed payloads or permanently revoked permissions usually will not. Use exponential backoff with jitter, respect provider retry guidance, and impose a maximum attempt count or elapsed retry window.

Give each attempt a deadline shorter than the queue's visibility timeout, or renew the lease using the queue's supported mechanism. A lease is not a perfect lock: network delays and paused processes can still produce overlap. For queue-specific behavior, review Amazon SQS visibility timeouts.

Set bounded concurrency according to downstream capacity. A hundred available workers do not imply that the database or email provider can handle a hundred simultaneous operations. Separate expensive exports from short notifications so that one workload cannot consume every worker slot.

Move repeatedly failing messages to a dead-letter destination with enough metadata to diagnose the cause. Replaying them should be a controlled operation after a fix, not an automatic loop. Preserve the logical job identifier during replay so that deduplication continues to work.

Operate the queue as a user-facing system

Track queue depth, oldest-message age, attempt counts, execution duration, and terminal failures. Oldest-message age often reveals user pain more directly than depth: ten stuck reports can matter more than a thousand fast notifications. Establish alert thresholds from the completion promises your product makes.

Include a correlation identifier across the accepting request, outbox row, message, and worker logs. Keep progress updates coarse enough to avoid excessive writes. Authorize status endpoints by tenant and ownership; a hard-to-guess job identifier is not an access-control policy.

Cancellation needs explicit semantics. A queued job can often be marked canceled before execution; a running job must check a cancellation flag at safe boundaries. Cancellation cannot unsend an email or reverse every external side effect. Explain these limits in the interface rather than displaying an unconditional cancel button.

Test crashes before acknowledgement, queue outages, invalid payloads, expired leases, and duplicate messages. Inspect sample messages with a JSON formatter and document operational decisions alongside your backend API resources. A reliable queue is not merely infrastructure: it is a complete lifecycle from acceptance to a visible, accountable outcome.

Key takeaways

• Accept work only after recording durable responsibility for it.
• Use an outbox to close the database-to-queue consistency gap.
• Expect duplicate delivery and make effects idempotent.
• Bound attempts, execution time, and downstream concurrency.
• Expose authorized job status and monitor completion delays, not just worker uptime.