fix(server): service launcher reports and aborts its pre-update database backup

[Silthus]

Problem

When the background service updates itself, the launcher stops the old server and copies the state database as its rollback snapshot before it starts the new one. On a headless Linux box with a multi-GB database, that copy took about 4 minutes. During the copy, three things went wrong:

• Silent: the launcher logged nothing after the handoff and the port did not listen, so the server looked dead.
• Not interruptible: systemctl --user stop got no reaction. systemd waited out TimeoutStopSec and then SIGKILLed the launcher.
• Restarts from zero: the next launch deleted the staging copy and began again. A user who restarted because T3 "hung" never let the update finish. Leaving it alone let it commit in about 4 minutes.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door (merge-risk rubric)

• Changes how the rollback snapshot is written: a streamed copy (with a clone attempt first) replaces fs.copyFile. A bad copy would leave a broken snapshot, and only a failed update would expose it, by restoring that snapshot.
• A stop during the snapshot now aborts it and starts no trial. That changes launcher stop sequencing, which the managed-tunnel cleanup depends on (see below).
• Limited by: pipeline resolves only after the write stream has finished, and each file is still fsynced before the staging directory is renamed into place. The existing snapshot and restore tests pass unchanged, and new tests cover the stop path. No state format changes, so a revert fully undoes it.

Change

The launcher now makes this step observable and stoppable:

• It logs before the snapshot (size, destination, target version), a heartbeat every 15 seconds (bytes copied and percent, then "flushing it to disk" during the final fsync), and the elapsed time at the end. A failed backup logs its cause before the launcher returns to the previous version.
• SIGTERM/SIGINT aborts the copy between 1 MiB chunks through an AbortSignal. The staging directory is removed, no trial starts, and the update stays pending, so the next launch retries it. The launcher exits cleanly instead of waiting for SIGKILL.
• The copy tries COPYFILE_FICLONE_FORCE first. That keeps the instant clone that copy_file_range already gave on Linux btrfs/XFS, which a plain streamed copy would lose. Everywhere else (ext4, and macOS, where the clone fails with ENOSYS as [Bug]: Server: service self-update stops the server before checking the database backup fits #14515 notes), it streams.
• The stop transition also clears the trial's prepared-timeout timer. Before this, a stop that landed while a trial was being spawned kept the launcher process alive for another 120s (visible in the "before" run below).

Trade-off: managed tunnel. Today, a stop during the handoff lets the trial start, and the trial releases a CLI-managed tunnel on its way down. When a stop now interrupts the snapshot, no trial starts, so that tunnel is kept for the next launch. Starting a trial without a complete snapshot would be worse, because its migrations could not be rolled back. The tunnel is already kept the same way today whenever systemd SIGKILLs a long backup. I added one sentence about this rule to docs/internals/server-updates.md.

Not in this PR, as follow-ups:

• Free-space check before the snapshot ([Bug]: Server: service self-update stops the server before checking the database backup fits #14515). The current ENOSPC path already removes the staging directory and rolls back. The fix there is to check before the server is stopped, which is a separate change to #handleUpdateRequest.
• Resumable snapshot. A stopped backup still starts over on the next launch. Now it is visible, so users know to wait.
• Staging directory fsync. This was there before this PR: the staging directory is not fsynced before it is renamed into place. After a power loss, a backup could be missing a file, and restore treats a missing file as "delete the live one". It is a one-line fix, but a separate durability problem.
• Unbounded database growth. Most of the size in this case was the append-only orchestration_events table. Bounding or compacting it would shrink the snapshot itself, and needs its own design.

Scope and approval

This is a focused reliability fix to one launcher step: the pre-update snapshot. There is no triaged issue for the silent, unstoppable copy itself. Related: #14515 (triaged: the same step, missing a free-space check) and #17937 (upgrade brittleness on large databases).

Verification

Test seams, both through the public Launcher (run(), stop(), and the persisted service state):

1. A stop during the pre-update backup keeps the update pending for the next start. The test starts a launcher with a pending update and stops it. It asserts that the update is still pending and that neither db-backup/<id> nor its .staging directory exists. Before the fix, it failed: the backup completed and the trial spawned anyway.
2. The launcher announces the pre-update backup and how long it took, through an injected log sink. Before the fix, it failed (timed out with no log line).

Checks run:

• vp test run apps/server/src/serviceLauncher.test.ts: 11 passed (9 existing, including database restore after a failed trial, plus 2 new)
• vp lint on both files, and vp run typecheck in apps/server: clean

Manual check: I ran the real launcher (main() with T3CODE_HOME set) on a pending update with an 8 GiB database on ext4, using fake runtimes that only idle, and sent SIGTERM from a shell.

SIGTERM 8 seconds into the copy, after the fix: exits in 1.0s, update still pending, no staging left behind.

11:07:53 [service-launcher] Backing up the database (8.0 GiB) to /tmp/launcher-proof/home/runtime/db-backup/proof-update before starting t3@1.1.0.
11:08:01 sending SIGTERM to launcher
11:08:02 [service-launcher] Database backup stopped; the next launch starts it again.
11:08:02 launcher exited with code 0 after 1.0s
update status: pending
db-backup contents: []

Same SIGTERM, before the fix: no output at all, and the launcher outlived the signal by 187s. The backup finished, the trial started anyway, and the trial's 120s timer kept the process alive.

11:02:51 sending SIGTERM to launcher
11:05:59 launcher exited with code 0 after 187.4s
update status: pending
db-backup contents: [proof-update]

A full backup, after the fix, showing the heartbeat through the final fsync:

11:10:13 [service-launcher] Backing up the database (8.0 GiB) to /tmp/launcher-proof/home/runtime/db-backup/proof-update before starting t3@1.1.0.
11:10:28 [service-launcher] Database backup in progress: 7.7 GiB of 8.0 GiB copied (95%).
11:10:43 [service-launcher] Database backup copied; flushing it to disk.
11:10:58 [service-launcher] Database backup copied; flushing it to disk.
11:11:13 [service-launcher] Database backup copied; flushing it to disk.
11:11:22 [service-launcher] Database backup finished in 69.7s.

Review: one cross-model pass (Codex, GPT-6.1 Sol, high effort) found no blocking defects. It flagged that a cloned file never counted toward progress (fixed in the follow-up commit), and the pre-existing staging fsync gap (listed above as a follow-up).

Gaps:

• The committed stop test stops before the copy starts. Aborting in the middle of the stream is shown only by the manual run above.
• The test file was sparse, so reads were nearly free. With a real database, the copy is disk-bound and the percent lines advance more slowly.
• A SIGTERM during the final fsync still waits for it to finish, because fsync cannot be cancelled. On this machine that took about 50s for 8 GiB.
• Not run on macOS or Windows. Both take the streamed path, because the clone attempt fails fast there.
• Not run under systemd itself. The shell sent the same SIGTERM that systemd would.

Model and harness: Claude Opus 5.5 in Claude Code, inside T3 Code.

🤖 Generated with Claude Code

[josephv123]

Additional ext4 data from the performance audits, consolidated here because this PR already identifies the offline database-copy step. No separate new issue is needed for the same update gap.

Ubuntu service, nightly 0.0.46-nightly.20261014.3132, roughly 1.46 GB database. Across retained updates, the gap from old-server shutdown reconciliation to the next Listening grew from roughly 2 seconds to a latest 6.5 seconds. A same-volume copied-database benchmark of cp --reflink=auto plus fsync took 6.6, 7.6 and 7.9 seconds. Those benchmarks ran under other audit/install load and are noisy; the production log cannot separate copy time from new-server boot.

Current upstream e536bfd33bc3c351997141d93790df75eeded7b9 confirms the ordering: #startTrial runs backupDatabaseOnce after the old child is dead and before the trial starts; on ext4 the clone attempt falls back to a full copy. So clone support does not remove this offline step on this filesystem, and progress/elapsed-time logging would make it diagnosable.

This does not establish that all measured downtime is copying, that a safe online-copy alternative is trivial, or that every historical update copied today's database size. The audit's initial estimate of ~30 GB of past backup writes extrapolated the present size to every update and should not be treated as a measurement. No live update or service restart was performed for this consolidation, and this is not validation of the PR's current head.