VEC-3 fix: recover model downloads after a failed transfer

[IchenDEV]

Summary

Fix model downloads that become unrecoverable after a failed, cancelled, or stalled transfer. Retries now use generation-isolated staging, reject late publication from cancelled writers, preserve complete published models, and keep expensive cleanup/candidate preparation off the MainActor.

The accepted product code and tests are fixed at 1cba5718aef0dd61254838fa77d8181cb4af820a. Later commits are documentation-only; no product-code change or full regression rerun is part of the remaining evidence follow-up.

Root cause

1. Concurrent or retried writers could use shared cache/download paths, allowing an old writer to corrupt or replace a newer generation.
2. Cancellation changed UI state before dependency I/O necessarily returned, so a late old writer could still attempt publication.
3. Hub snapshots could publish symlinks back into disposable staging caches.
4. Candidate preparation, rollback backup creation, and recursive cleanup could block the MainActor.
5. Abnormal exits could leave managed generation, promotion, backup, and retired-cleanup artifacts behind.

Fix

• Give Whisper, LLM, and ASR downloads a token-scoped .utter-generations/<token>/download root and isolated Hub cache from operation start.
• Serialize token validation, publication, Cancel, and Delete arbitration; reject every late write from a retired generation.
• Materialize Hub symlinks into regular files before deleting staging and atomically publish prepared candidates with rollback restoration.
• Prepare candidates/backups and delete retired trees in detached work; keep only short same-volume arbitration on the MainActor.
• Sweep all managed orphan artifacts at startup through the real ModelCatalog initializer path, off the MainActor, and gate fresh download entry points until it completes.
• Allow a replacement ModelCatalog.downloadWhisper request to start while the cancelled dependency call is still suspended, without sharing its staging root or publication rights.
• Retain the 120-second no-progress watchdog and distinguish user cancellation from transport failure.

Fixed code identity

• Code commit: 1cba5718aef0dd61254838fa77d8181cb4af820a
• Whole tree: d15b2c7b02262bf4823646903730122dba90a56f
• Sources tree: ca355f4ff185ece0cc99a3c73cb9921510018ccc
• Tests tree: 7d69e1226b6a109bffc5ae3c3b0a83dace33c9d3
• Current materials baseline: 263071e0885fc0e9245c35794e850fca2c76dadb (documentation only; Sources/Tests identical to 1cba5718)
• Follow-up documentation commit produced from the review patch: 249eca3fd79df4bced0ec42c55baba7dcb383e47 (documentation only; Sources/Tests identical to 1cba5718)
• Toolchain-provenance correction: bf859aa3139c54b0e560ed9123a116407d1c9ee8 (documentation only; Sources/Tests identical to 1cba5718)
• Raw-timing correction: 47a0c276e4abf80f76c935c282dcbcda13e6329c (documentation only; Sources/Tests identical to 1cba5718)

Accepted macOS verification at 1cba5718

• Raw-log toolchain: macOS 27.2, Xcode 27.0 (build 27A266a), Apple Swift 6.4 (swiftlang-6.4.0.34.1, clang-2100.3.34.1).
• swift build: exit 0.
• Focused P1 counterexamples: 3 executed, 0 failures, including:
  • testModelCatalogInitializerStartupCleanupKeepsMainActorResponsive
  • testStartupCleanupKeepsMainActorResponsive
  • testApplicationCatalogResumeEntryStartsBeforeOldDependencyReturns
• Full suite: 670 XCTest executed, 14 skipped, 0 failures.
• Skip taxonomy: 4 cases require OPENTYPE_LIVE_DOWNLOAD_INTEGRATION=1; the other 10 are environment/model gated (ANE model, Apple Speech sample, chat-template probe, foreground bundle identity, Espresso fallback, prompt dump, Qwen native ASR ×2, streaming ASR ×2).
• Release-style bash scripts/build-app.sh --app-only: exit 0; signed dist/Utter.app with a 3.7 MB default.metallib.
• Live Whisper interruption → Resume → atomic commit → model load → real transcription: exit 0 (docs/assets/demos/en-sample.m4a, 0.15 seconds).
• Historical live Hub evidence remains attributed to its recorded source SHA: full 278 MB weights, symlink materialization, tokenizer round trip, MLX container load, and real text generation.
• Existing mutation summary: synchronous initializer cleanup exited 1; Task { @MainActor in } cleanup exited 1; the restored test exited 0 after each mutation and returned to tree d15b2c7b….

Raw accepted regression log: mac-p34-1cba5718-verification.log.

Audit-grade mutation transcript

The audit-grade mutation transcript was captured on macOS using mac-p102-mutation-harness.sh. It created an independent clean checkout fixed at 1cba5718aef0dd61254838fa77d8181cb4af820a, performed exactly four focused runs, streamed and recorded complete combined stdout/stderr, verified target assertion failure markers, and verified exact tree restoration.

Its raw header records macOS 27.2, Xcode 27.0 (build 27A266a), and Apple Swift 6.4 (swiftlang-6.4.0.34.1, clang-2100.3.34.1). The Xcode 16.2 / Swift 6.0.3 claim in handoff prose was a transcription error, not execution evidence. Earlier live-download and model-loading results retain their original commit and log attribution below.

The four focused runs were:

1. Mutation A: initializer performs synchronous cleanup — failed at UtilityTests.swift:409-413 (5 expected failures).
2. Restore A exactly — same focused test exited 0 (passed in 0.033s) and tree equaled d15b2c7b02262bf4823646903730122dba90a56f.
3. Mutation B: cleanup uses Task { @MainActor in } — failed at UtilityTests.swift:411-412 (2 expected failures).
4. Restore B exactly — same focused test exited 0 (passed in 0.032s) and tree equaled d15b2c7b02262bf4823646903730122dba90a56f.

Measured results (mac-p102-mutation-harness.log):

• Log SHA-256: 5b6ea6e1d45fec7a56d60513023ab0c3e305e6f152398b98ad3c68660e484240
• Mutation A exit / target assertions / elapsed: exit 1 / failed with 5 expected assertion failures at UtilityTests.swift:409-413 / elapsed 268s (includes initial scratch build).
• Restored A exit / exact tree / elapsed: exit 0 (passed in 0.033s) / matched exact baseline d15b2c7b02262bf4823646903730122dba90a56f (git diff --exit-code 0) / elapsed 14s.
• Mutation B exit / target assertions / elapsed: exit 1 / failed with 2 expected assertion failures at UtilityTests.swift:411-412 / elapsed 14s.
• Restored B exit / exact tree / elapsed: exit 0 (passed in 0.032s) / matched exact baseline d15b2c7b02262bf4823646903730122dba90a56f (git diff --exit-code 0) / elapsed 12s.
• Final full-tree diff against 1cba5718aef0dd61254838fa77d8181cb4af820a: exit 0, 0 harness failures, porcelain status clean.

This evidence recapture does not require another product-code build, full XCTest run, live-download run, or Release build.

Evidence lineage

• 0167504b920de6b36cb36f1b8e703ce486d69e8f: baseline recovery implementation.
• 28a847f6018871f70210f119c4d650aed1e13040: live Whisper/Hub interruption and Resume verification.
• 6bcc5adc69c42f21aa905c05b74b6e041fc0b4c7: detached candidate preparation, rollback, and startup cleanup integration.
• 522947265e83f07c730b00c2130db5c45727de55: MLX container load and real generation with default.metallib.
• 3442f83641869f912247bef796e624efd813605a: managed-artifact reclaim, O(1) retirement, detached delete, and token recheck.
• eb4c08a39b2104cbb1045fe0cd7eb589bd1c8e3f: historical documentation baseline before the final P1 continuation.
• 1cba5718aef0dd61254838fa77d8181cb4af820a: accepted final code/test tree and macOS verification target.
• 332e38dafabbe46f3514514a21aba5fe7163ce5d: initial documentation of the measured p3/p4 results.
• 263071e0885fc0e9245c35794e850fca2c76dadb: documentation-only rerun totals, skip taxonomy, mutation summary, and resource-risk decision point.
• 249eca3fd79df4bced0ec42c55baba7dcb383e47: documentation-only follow-up recording the audit-grade mutation transcript and measured results.
• bf859aa3139c54b0e560ed9123a116407d1c9ee8: documentation-only correction of the raw-log toolchain provenance and stale pending wording.
• 47a0c276e4abf80f76c935c282dcbcda13e6329c: documentation-only correction of the Restored A timing to the raw-log value.

Residual risk and release gates

• Release-owner decision: if underlying URLSession/I/O never returns, a cancelled transfer retains its background Task and staging root. Repeated permanent hangs can accumulate tasks and staging directories until process restart. Write isolation and immediate retry remain intact, but the release owner must either accept this dependency-layer risk or require a resource ceiling/process isolation.
• The 120-second stall threshold is a product-visible heuristic and remains subject to reviewer approval.
• Intent/spec/verification approval, final documentation review, external CI, licensing, and publication approval remain required.
• Do not merge or publish until the release-owner decision and human review gates are complete.

---

Fixes VEC-3