Skip Redis for the job queue, then meet SQLite's four walls
SQLite in WAL mode is a legitimate background-job queue on one box, but WAL doesn't remove the single-writer limit — it relocates it. Here are the four ways that limit bites, three with a fix and one with only a migration.
You do not need Redis to run background jobs. In 2026 that is not a contrarian take — it is close to the default. Solid Queue ships as the Rails 8 Active Job backend, backed by a plain database table. Python has litequeue, Node and Bun have PlainJob claiming 15,000 jobs/sec single-process, Go has liteq with jobs that survive a restart. SQLite's own maintainers will tell you a site under 100K hits/day should be fine on SQLite, and they mean it conservatively. A jobs table gives you durability, inspectability with SELECT, and zero extra infrastructure to run, monitor, and page someone about at 3am.
All of that is true. The part that gets skipped is where it stops being true. Most posts on this end at "WAL lets readers and writers coexist, so ship it." WAL does not remove SQLite's single-writer limit. It moves where that limit bites you — and it bites in four qualitatively different ways, each with its own trigger. Three of them have a fix. The fourth is a wall.
What WAL actually buys you
Write-Ahead Logging, on by default since SQLite 3.7.0 in 2010, changes one thing that matters here: readers and writers no longer block each other. An unlimited number of readers can run against the database while a single writer appends to the WAL. Before WAL, a writer locked everyone out. After WAL, your worker can claim a job while a dashboard reads the queue depth, and neither waits on the other.
What WAL does not change: there is still exactly one writer at a time, always. It removed reader/writer contention. It did nothing to writer/writer contention. Everything below is a consequence of that single sentence.
Claiming a job, and why the race can't happen
The atomic claim is one statement, available since the RETURNING clause landed in SQLite 3.35.0 (March 2021):
UPDATE jobs
SET status = 'running', locked_at = unixepoch()
WHERE id = (
SELECT id FROM jobs
WHERE status = 'queued'
ORDER BY id
LIMIT 1
)
RETURNING *;
Postgres needs FOR UPDATE SKIP LOCKED here so that N workers hitting the table at once each grab a different row instead of fighting over the same one. SQLite has no equivalent, and it does not need one — because two workers cannot execute this UPDATE at the same instant. The write lock serializes them. Worker A claims the row and commits; worker B's identical statement then runs against the already-updated table and picks the next row. The race isn't handled cleverly. It's impossible by construction. That is the whole trick, and it is genuinely good: you get correct claiming with no SKIP LOCKED, no advisory locks, no Lua.
The cost of that serialization is what the four failure modes are made of.
Failure mode 1: the busy_timeout lie
Everyone sets PRAGMA busy_timeout = 5000 and assumes contention now just means "wait a bit." It does not always mean that. If a connection opens a read transaction and then tries to upgrade to a write mid-transaction, SQLite does not wait out the timeout — it returns SQLITE_BUSY immediately, ignoring busy_timeout entirely. It has to, to avoid violating serializability: another writer may have already changed the pages this transaction read.
The queue-flavored version of this bug is the check-then-claim written as two statements: SELECT a queued job, then UPDATE it in the same transaction. Under any concurrency that intermittently throws "database is locked" despite your timeout, and it looks like a timeout that isn't working. The fix is to declare write intent up front with BEGIN IMMEDIATE, or to issue the write statement first. The single UPDATE ... RETURNING above sidesteps it because it never reads before it writes.
Failure mode 2: checkpoint starvation
The WAL is not the database. Periodically SQLite checkpoints it — copies committed pages back into the main file and lets the WAL be reused. The default auto-checkpoint fires around 1000 pages, roughly 4MB. Here is the catch: a checkpoint must stop when it reaches a WAL page past the read mark of any current reader. One long-lived reader pins the checkpoint in place.
So a single stray reader — an admin dashboard connection, a monitoring query, an ORM session someone forgot to close — can stall the checkpointer no matter how healthy your job throughput looks. One documented incident: a forgotten open read transaction on a reporting connection let the WAL grow from a 2GB database to 20GB in a few hours, at roughly 100MB/min. Nothing was wrong with the writes. The checkpointer simply could never finish. The mitigations are concrete: PRAGMA journal_size_limit = 67108864 to cap reclaimed WAL size, drop wal_autocheckpoint from 1000 to 100 pages so checkpoints attempt more often, tighten connection lifecycle so nothing holds a read transaction open, and — most importantly — alert on WAL file size directly. It is a leading indicator no request-latency graph will show you.
Failure mode 3: the throughput and fairness ceiling
The single writer has a real number attached. On a 1KB record workload with WAL and synchronous=NORMAL, one benchmark measured about 70k ops/sec at one thread, holding near 67k across 2–64 threads, sagging to 62k at 128, and throwing errors at 256. Notably, wrapping writes in an application-level mutex — letting your process serialize what SQLite was going to serialize anyway — held steady near 60k ops/sec from 1 to 256 threads with no lock errors. Serialize deliberately and you beat the free-for-all.
That is degradation. The cliff is separate. SkyPilot pushed one SQLite file with 1000+ simultaneous processes and hit repeated "database is locked" crashes despite WAL. Median write latency climbed to 2.3 seconds, with a geometric tail — 0.13% of writes took over 60 seconds. And because SQLite's busy-retry backoff has no FIFO fairness, a process that just arrived can win the lock over one that has been waiting far longer. That is writer starvation: not slow-but-fair, but a livelock where the unlucky job is starved indefinitely. It degrades gracefully, and then it falls off a cliff.
Failure mode 4: the wall, not a cliff
The first three have fixes. This one has only a migration. Every process using a WAL database must be on the same host. WAL does not work over a network filesystem — NFS and SMB byte-range locking is unreliable by design, and the SQLite docs say so plainly. So the moment your workers need to run on more than one machine, you are done. Not slow — architecturally over.
And the usual escape hatches don't apply. Litestream and LiteFS-style replication is physical and one-directional: only the primary writes, and writing to a replica corrupts it. Reads scale out; writes stay pinned to one box. The experimental BEGIN CONCURRENT fork uses optimistic page-level locking for concurrent writers but suffers heavy false-conflict rates on auto-increment/rowid keys and time-ordered indexes — precisely the shape of a jobs table. Turso is rewriting SQLite in Rust to add MVCC and lift the single-writer limit, which is itself the tell that the limit is fundamental; that MVCC support is explicitly experimental, not production-ready. There is no PRAGMA for multi-host. It is a rewrite.
Where the line actually is
Single box, bursty or moderate job volume: use the SQLite table. It is arguably better than Redis for this — the jobs are durable across restarts, you can debug the queue with SELECT, and there is no second system to operate. Put the queue in its own SQLite file, separate from your application database, exactly as Solid Queue does; it keeps queue write contention off your app's reads, and Solid Queue is candid that the lack of SKIP LOCKED means worker processes queue up for the write lock.
Move off SQLite when either boundary is in view: sustained thousands of writes per second, or workers on more than one host. The first is a tuning-and-hardware conversation — Postgres with FOR UPDATE SKIP LOCKED gives you true parallel claiming, or reach for a real broker. The second is not a conversation at all.
Skipping Redis is a good default in 2026. Watch the WAL size, claim with one write statement, and know your ops/sec ceiling before you meet it. But skipping Postgres once you need more than one writer host is not a PRAGMA you forgot to set. It is a rewrite you scheduled by not deciding — and the cheapest time to see it is now, not during the incident.
