Skip to content

Back up and restore an instance (spec 030, #41) - #124

Merged
plainprogrammer merged 35 commits into
mainfrom
030-backup-restore
Oct 23, 2026
Merged

Back up and restore an instance (spec 030, #41)#124
plainprogrammer merged 35 commits into
mainfrom
030-backup-restore

Conversation

@plainprogrammer

@plainprogrammer plainprogrammer commented Oct 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Closes #41. Spec 030 (spec, v1.2.1; plan; verification); ADR 0024.

What it does

An operator backs up a running instance and restores it onto a fresh one, following a documented and tested procedure. This is the self-hoster's own backup; the per-collector export (specs 022, 025, 028) is unchanged, and the README says which to use when.

  • Back up, online. bin/rails "backup:create[DIRECTORY]" takes a consistent snapshot of the primary database with SQLite's VACUUM INTO while the app keeps writing (ADR 0024). The snapshot is checked (integrity and foreign keys) before it counts, written as one collector-backup-<UTC time>.tar.gz (a manifest plus the database) under a .partial name, mode 0600 from creation, flushed to disk and linked into place; a final name is never overwritten. The destination is required and is never inside the volume it protects. The command refuses to start without room for the snapshot and the archive.
  • What a backup holds (ruled): the primary database, whole, catalog included. Not Active Storage files (only import uploads that expire within an hour; a suite guard fails when a collector-owned attachment is added), not catalog downloads or art images (rebuilt; the docs say to press Build art index), not the queue, cache or cable databases, and never SECRET_KEY_BASE or RAILS_MASTER_KEY.
  • Restore onto a fresh volume. bin/rails "backup:restore[FILE]" runs before the instance's first start, as a one-off, and refuses a volume holding any database or -wal/-shm/-journal file. It never opens the primary's path or runs a migration; it places the snapshot in rollback-journal mode after every check passes, so a failed or killed restore leaves nothing that the next boot's db:prepare would mistake for an empty instance. It clears sessions (everyone signs in again; their cascades are guarded). A backup from a newer release is refused; an older one is migrated by db:prepare on first start.
  • Plain refusals for a file that isn't a backup, an incomplete or damaged one, a wrong release, no room, a non-fresh volume, or an unreadable file — never a backtrace.
  • No secret needed to restore. The tasks don't boot the app, so they read no secret_key_base; a production-boot spec proves it in CI.
  • Documented for Compose, Podman and Kamal (README "Backup and restore"), with copy-in and uid 1000 ownership per engine, a cron line, and a manual recovery path using only tar and sqlite3. Kamal gets a backup alias and a commented backups volume with its install -d step (ruled); its restore sequence is a one-off docker run of the exact release image after kamal server bootstrap (ruled), documented and not exercised through Kamal itself.
  • bin/backup-smoke, a new step in the image job on both architectures after bin/image-smoke: back up a running container, copy the file out, restore it into a fresh container under a different SECRET_KEY_BASE, and check /up, sign-in, the lots and the recurring tasks. It takes an optional second image so Prove an upgrade from the previous release in CI #42 can reuse it.

Evidence

Check Result
Backup, verification included (bar < 5 min) 1.7 s for a 121 MB seeded database (100,000 catalog entries, 50,012 lots); archive 8.6 MB
Writes while the backup runs (bar: each under the 5 s busy timeout) 12 writes, longest wait 0.013 s
Restore (bar < 5 min) 1.8 s
bin/backup-smoke passed locally with podman on linux/arm64 against an image built from the branch; amd64 is this PR's CI
bin/ci on this branch every step passes; 1865 examples, 0 failures, 1 pending (art_agreement_spec, needs COLLECTOR_ART_AGREEMENT); also passed in the pre-push hook

The measurement used a seeded database under tmp/, not a real instance (ruling m7).

Reviews

  • Spec: two independent Opus reviews (v1.0.0, then v1.1.0), every decision ruled by the maintainer, approved at v1.2.1.
  • Plan: independent Opus review (1 critical, 3 major, 13 minor), the maintainer's rulings, then a second Sonnet review (approve with nits, folded in); every finding is mapped in plan-review.md.
  • Implementation: independent Sonnet review (Mode B), 7 minor findings fixed or recorded. Verdict in verification.md: SPEC-ALIGNED (independent Sonnet review 2026-10-22, after fixes).

Still owed

🤖 Generated with Claude Code

@plainprogrammer plainprogrammer self-assigned this Oct 22, 2026
@plainprogrammer plainprogrammer mentioned this pull request Oct 23, 2026
Merged
@plainprogrammer plainprogrammer mentioned this pull request Oct 23, 2026
Merged
plainprogrammer and others added 26 commits October 23, 2026 19:42
@plainprogrammer @claude
docs(specs): draft spec 030, back up and restore an instance
62dc982 
Issue #41, roadmap theme A item 6. Decisions 1 to 3 (mechanism, contents,
secrets and sessions) ruled by the maintainer; 4 to 14 are recommendations.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(specs): revise spec 030 after the Mode A review (1.1.0)
55b239e 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(specs): revise spec 030 after the second Mode A review (1.2.0)
4d31c8b 
Records the maintainer's 2026-10-21 rulings (decisions 1 to 15; the Kamal
restore sequence as option (a)) and fixes M1 to M4 and m1 to m11 of the
round 2 review; review.md maps each finding.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(specs): approve spec 030, back up and restore an instance (1.2.1)
ae3f1c9 
The maintainer approved spec 030 on 2026-10-21 with rulings on decisions
1 to 15 (4 to 15 as recommended, the Kamal restore sequence as option (a));
both Mode A reviews' findings are fixed. Status and index only, no change
to any requirement.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(adr): propose ADR 0024, back up with VACUUM INTO and restore ont…
90a9660 
…o a fresh volume

Records the backup mechanism spec 030 rules (decisions 1 to 3, 5 to 7) and the
plan's placement of the code in lib/collector/backup, with tasks that don't
boot the application. Status Proposed until the maintainer approves the plan.
Numbered 0024: origin/main holds up to 0022 and 029-manage-locations takes 0023.
The index loses the backup mechanism from its Expected line.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(specs): plan spec 030, back up and restore an instance
40cec8a 
Nine phases in one pull request: the namespace and guards, the file
(manifest and archive), the SQLite snapshot, backup:create, backup:restore, the
restored instance and upgrade proofs, bin/backup-smoke with its CI step, the
documents, and the measurements. With the contracts for the two tasks and the
archive, the data-model note (no schema change), research (the secret_key_base
trace: tasks that skip :environment read none), and the draft of the separate
art-index issue for the coordinator. Not reviewed or run yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(adr): say ADR 0024's archive is linked into place, not renamed
6ae2941 
The plan publishes by hard link then unlink (a rename only where a share can't
link), which is what makes "never replace a file" hold. The Proposed status stays.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(specs): revise the spec 030 plan after the Opus review
59c334d 
Fixes the critical (Create's read-back refused its own .partial archive), the
three major findings (the D4 boot spec's wrong master key, Phase 0's encryption
guard under load_defaults 8.1, an unreadable file or unwritable volume ending in
a backtrace) and minors m1 to m13, with the maintainer's rulings of 2026-10-21:
the room is checked before the checksum (m3), the Kamal volume stays commented
out (m5), AC-3.6's construction is accepted (m6) and the measurement runs on a
seeded database outside storage/ (m7). Also fixes the expect_refused sentence
check, which refused every message that starts with a file name.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(specs): record the spec 030 plan review and its resolutions
12b5f77 
Maps the Opus review's C1, M1 to M3 and m1 to m13 to their fixes and commits,
with the maintainer's rulings of 2026-10-21 (m3, m5, m6, m7).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(specs): fold in the second review of the spec 030 plan and recor…
a3b082b 
…d its approval

The Sonnet review (APPROVE WITH NITS, 4 minor) is mapped in plan-review.md:
the catch-all line no longer claims nothing was changed, expect_refused keeps
its capital rule, the m3 residue is stated in the map row and choice 3, and
Phase 5 says the boot examples first run in CI. The maintainer approved the
plan on 2026-10-21 and chose Sonnet for execution.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
feat(backup): add the backup namespace, the volume's definition and t…
9a282dd 
…he suite's guards

Plan step: Step 0.8: Commit feat(backup): add the backup namespace, the volume's definition and the suite's guards

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
fix(backup): write the guards spec of Step 0.6
bd6336e 
Plan step: Step 0.6: Write the guards. The extraction script had left the file empty.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
feat(backup): add the backup file's manifest and archive
22f57d6 
Plan step 1.6 (spec 030 Phase 1). Create_private now sets binmode and the
archive reads one byte past the end so zlib checks the gzip footer.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
feat(backup): add the snapshot's SQLite operations
b04bdcf 
Plan step 2.5 (Phase 2). Snapshot owns every SQLite call of both commands. OrphanRows counts rows, not broken keys.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
feat(backup): add backup:create
9a18951 
Plan Phase 3, step 3.5: Backup::Create, the backup:create task and their specs (AC-1.1 to AC-1.9, AC-1.11, AC-1.12, AC-4.1).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
feat(backup): add backup:restore
7894951 
Plan Phase 4, steps 4.1 to 4.6: Restore places a verified snapshot on a fresh
volume through a temporary file linked into place, ends every session, and
refuses with two plain sentences; the backup:restore task.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
test(backup): prove the restored instance, the upgrade, the boot and …
2184191 
…the guards

Plan step 5.5 (Phase 5): AC-2.2, AC-2.4, AC-2.8, AC-2.9, AC-2.10, AC-3.6, AC-6.2,
FR-5 and the D4 boot examples; adds with_restored_instance (step 5.1).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
test(backup): add bin/backup-smoke and run it in CI after the image s…
56368a7 
…moke test

Plan step 6.7 (steps 6.1 to 6.5): the smoke script, its CI step after the
image smoke test, the Kamal backup alias with a commented-out backups volume,
and keeping the script out of the image (checked by bin/image-smoke).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(backup): document backup and restore, the file layout and the re…
32305df 
…lease check

Plan Phase 7, steps 7.1 to 7.8: the README's Backup and restore section and pointers, docs/backup-restore.md, the release check and upgrade note in docs/releasing.md, CLAUDE.md, roadmap item 6 and the SQLite rule.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
feat(backup): add the backup measurement script
ee68839 
Plan step 8.1: seeds a database under tmp/measure and times a backup, a
restore and the longest write wait (decision 13, AC-1.10).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(adr): accept ADR 0024, back up with VACUUM INTO
65e5910 
The maintainer accepted it with the spec 030 plan on 2026-10-21 (plan step 8.5).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(backup): record the measurements and verification of spec 030
dfa19d5 
Plan step 8.3: evidence per criterion, the seeded-database measurements,
the local smoke run, bin/ci twice. The NAS restore (AC-5.6) stays owed to
the maintainer; the implementation review is not yet recorded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
test(backup): run the AC-1.2/1.3 writers in child processes, pinned t…
8a80cfe 
…o the VACUUM INTO

The writers were threads in the backup's own process, with a window over the whole run, so
a backup that blocked them for a second would still pass. They are now two child processes
with connections of their own, and the example counts the commits that finished between the
new :snapshot_starting and :snapshot_written checkpoints (review minor 1).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
test(backup): make AC-2.9's run interrupted only because its job is gone
3a2bd8b 
The run's heartbeat was 20 minutes old, so the page read it as interrupted by the
15-minute stall rule whatever the restored queue held. At 5 minutes only the
JOB_ENDED_GRACE rule for a job the queue no longer has can say so (review minor 2).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
fix(backup): run the copied-in restore with SECRET_KEY_BASE_DUMMY, as…
acdc666 
… AC-5.2 says

bin/backup-smoke's copied-in pass and the README's copied-in command ran the restore
with no variable at all; both now carry SECRET_KEY_BASE_DUMMY=1 like the host-directory
pass (review minor 4, maintainer ruling Q2).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
test(backup): guard that every foreign key to sessions cascades on de…
0ee9a1c 
…lete

The restore ends sessions with DELETE FROM sessions and relies on the cascade for
bulk_removals and bulk_selections; a table with a plain key to sessions would make it stop
with a constraint error. The guard names spec 030 decision 3 and has a negative control
(review minor 5, maintainer ruling Q5).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
plainprogrammer and others added 9 commits October 23, 2026 19:42
@plainprogrammer @claude
fix(backup): call a file cut inside the gzip header incomplete
1820ec8 
A copy of 2 to 9 bytes starts like gzip and stops, which the contract classifies as
incomplete; GzipReader raised on the short header and it was reported as not a backup
(review minor 6).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
fix(backup): restore removes only its temporary file and that file's …
022668b 
…-journal

Backup.remove deleted a -wal and -shm as well; FR-6 names only the journal for
backup:restore. The suffixes are now an argument, and the restore passes its two
(review minor 7).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(backup): point the drafted art-index issue at #122, where it was…
705ae3b 
… filed

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(backup): record the implementation review of spec 030 and its fixes
ea1c9eb 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(architecture): distil spec 030 (no area doc yet; plan retirement…
76a1040 
… owed)

Spec 030's areas are release and deploy and data portability, which have
no area doc yet (#109, #108), so nothing is written to docs/architecture/.
The spec supersedes nothing in a shipped spec. The index row's Plan is
"retirement owed" and its status names the implementation review and the
NAS restore still owed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(030): record the distillation
ba4dcd4 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(backup): the export is format version 2 and holds locations (spe…
0c77bcf 
…c 029)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(backup): cite the rebased commits and record the rebase onto spe…
757bc38 
…c 029

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer @claude
docs(backup): cite the commits rebased onto the crass bump
f2dbedc 
The rebase onto origin/main at 980bff9 (#126) rewrote every commit of the
branch; the 16 SHAs the spec's documents cite are mapped to the new ones,
and verification.md records the rebase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plainprogrammer
plainprogrammer merged commit 4cd1ef5 into main Oct 23, 2026
4 checks passed
@plainprogrammer
plainprogrammer deleted the 030-backup-restore branch October 23, 2026 23:55
plainprogrammer added a commit that referenced this pull request Oct 24, 2026
@plainprogrammer @claude
Merge origin/main into 029-retire-plan for spec 030 (#124)
8ca7150 
The specs index conflicted on neighbouring rows: this branch's 029 row
(Shipped, plan retired) is kept, with main's new 030 row after it. Merged,
not rebased, so the retired plan's commit 88a52cf keeps its sha
(docs/git-convention.md, Merging).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 24, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@plainprogrammer plainprogrammer

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Back up and restore an instance

1 participant

@plainprogrammer