firedrill takes a Postgres backup and actually restores it into a disposable, version-matched container — then proves the result is usable. It fails your build the day a backup stops being restorable, not the day you need it.
A verification that could not run never reports as passing
A rung that could not run reads n/a, never a tick
It is always one of these, discovered at the worst possible moment. Every one is invisible to a backup job and obvious to a backup restore. That gap is the entire product.
| What went wrong | Why nobody noticed |
|---|---|
| pg_dump exited 0 but a full disk truncated the output | exit code 0 |
| The dump was of a replica that had stopped replicating | it ran nightly, without error |
| An extension (postgis, pgvector, citext) is absent on the restore host | never restored anywhere else |
Roles named by OWNER TO do not exist on the target | pg_restore warns and continues |
| The glibc/ICU collation differs, so text indexes sort differently | wrong rows, no error at all |
A sequence sits below max(id) | first insert after failover collides |
| Restore takes 14 hours; the stated RTO is 1 hour | never measured |
Each rung is independently reportable, and a rung that cannot run says so rather than passing. That distinction is the whole design: a tool that says OK because it silently skipped the restore is worse than no tool, because it manufactures confidence.
Local path, presigned URL, or S3. Size and checksum are verified before anything is restored — a dump that arrives truncated but plausible never reaches a container.
Into a disposable Postgres of the dump's own major version, read out of the archive header in pure Python. pg_restore's stderr is classified; warnings are findings, not noise.
The restored catalog against a committed reference — one line per object, sorted, diffable in review.
Row counts against a floor, or against the last known-good restore. A table that lost 90% of its rows restores perfectly.
Your own smoke queries. “There is at least one order from the last 7 days” catches a stale-replica dump that every structural check passes.
The checks only a restore can make: sequences ahead of max(id),
and collation — the silent one that returns wrong rows with no error.
Every stage timed against a stated RTO budget, and measured against the last good run. RTO becomes a trend you have measured, not a number you claimed.
Each of these changed the design. They are here because reasoning about Postgres gets you a plausible answer, and a restore tool cannot ship a plausible answer.
Not a different one — none. So an alpine target cannot tell you whether text indexes sort like production's, and neither can Postgres. That is two findings, not one.
postgres:16 2.41
postgres:16-alpine (empty)
A schema reference taken on 16 called every not-null column “drift” on 18. The column line already records not-null, so those rows are a duplicate representation — excluded, and one reference is now portable across majors.
+ constraint|public.customer.customer_email_not_null|n + constraint|public.customer.customer_id_not_null|n
2,000 rows arrive and the sequence stays at 1. Checking sequences under the sample tier would report a failure about firedrill's own sampling rather than about the backup — the most damaging kind of false positive.
rows 2000 · sequence 1
On a CI runner with an unreachable daemon. The availability probe trusted the exit code and called a dead daemon usable — the exact bug this tool exists to complain about, in its own front door.
rc=0 ServerVersion=""
Measured on both majors, custom format, in every broken case. The exit code is still insufficient — one bit never says which failure — so stderr classifies and the code is a backstop.
pg_restore: warning: errors ignored on restore: 1
exit 1
pagila links every sequence through the column DEFAULT, with no
OWNED BY anywhere — and pg_get_serial_sequence
resolves none of them. The sequence check examined nothing and said so as
“0 sequence(s)”, which reads like “fine”. Found by restoring
databases this project did not write.
present 13 · linked 0 → now 9 of 13
pagila's own schema defines rental_by_category as a
materialized view. Neither pagila-schema.sql nor
pagila-data.sql ever refreshes it, so a stock, unmutated
restore genuinely holds zero rows there — a query against it raises on
every fresh load, healthy or not. Found the same way as the sequence
bug above: restoring a database this project did not write, not by
reasoning about one it did.
MATVIEW_UNPOPULATED 1 materialized view(s) hold no data
Because bash there is the WSL stub. A failure with no message
at all is the tell. Probe a tool before trusting its exit code — the same
lesson, twice, in one project.
bash -n <<< ':' → rc=1, stderr=""
The synthetic corpus is written by the same person as the checks — precise, and unable to find the case its author did not think of. This breaks a real, unrelated 70-table schema one way at a time instead: drop a trigger, a function, a view, an index, a column; empty a table; walk a sequence back; backdate a year of rows; loosen a constraint; enable row-level security. Every mutation produced a finding, and the one restore nobody touched stayed silent.
11 mutation(s) · 11 caught · healthy: silent
That is measured, not claimed. pg_restore is perfectly happy with
all of them — which is the entire argument for the ladder. Each is asserted
in both directions, because a false positive costs exactly what a
false negative costs: a tool that cries wolf gets muted, and a muted tool is
worse than none because it still looks like coverage.
| Backup | What is wrong with it | Caught by |
|---|---|---|
| healthy | nothing | — exit 0 |
| truncated_data | cut off mid-restore | ARCHIVE_TRUNCATED |
| truncated_header | unreadable header | ARCHIVE_UNREADABLE · no container started |
| missing_role | OWNER TO a role that is absent | ROLE_ABSENT |
| missing_extension | hstore absent from the target image | EXTENSION_ABSENT |
| empty_database | a dump of the wrong database | EMPTY_RESTORE |
| missing_index | one index gone; data intact | STRUCTURE_MISSING |
| volume_drop | 99% of rows deleted before the dump | VOLUME_BELOW_MINIMUM |
| stale_replica | schema and counts perfect, data a year old | SEMANTICS_FAILED |
| sequence_behind | setval below max(id) | SEQUENCE_BEHIND |
| collation (musl target) | text indexes sort differently | COLLATION_MISMATCH |
No Postgres client on the host: the archive header is parsed in pure Python and every database operation happens inside the target container, which is what makes the version matching real.
Locally · v0.1.0 on PyPI
$ pip install firedrill $ firedrill run backups/nightly.dump [ok ] fetch 13,572 bytes [ok ] restore exit 0 [FAIL] semantics 1 check [n/a ] structure no reference configured FAIL — recent orders exist: expected > 0, got 0
In CI
- uses: MaXiMo000/firedrill@v0 with: config: firedrill.yml rto: 45m history: firedrill-history.json
Fails the build when the drill fails and when it could not run at all — “we did not verify” must never be quieter than “we verified and it was fine”. Publishes the report before failing, so a red build is an actionable one.
finally.