rust mode: superseded_by_memory_id silently lost when authority is dropped with pending mirror references outstanding

[iceteaSA]

Summary

The mirror projection nulls memories.superseded_by_memory_id as a deliberate transient placeholder, then restores it either immediately or via a deferred pending-reference translation. That translation is reachable only from applyMirrorPage. If rust mode is deactivated while pending references are outstanding, the translation never runs, the NULLs become permanent, and updated_at is never bumped — so the loss is invisible to any audit that keys on modification time.

Observed on a live store: 61 valid supersede pointers destroyed in a single activate/deactivate cycle.

Mechanism

applyMemoryRow writes NULL first (context-authority.ts:1598):

hasSuperseded ? null : (existing?.superseded_by_memory_id ?? null),

then attempts translation (:1609-1626):

if (hasSuperseded && typeof row.superseded_by_memory_id === "number") {
    const translated = mirrorIdentity(db, "memories", moduleProject, row.superseded_by_memory_id, statements);
    if (translated) {
        statements.updateSuperseded.run(translated.context_row_id, contextId);   // restored
        statements.deletePendingReference.run(moduleProject, feed.module_row_id);
    } else {
        statements.upsertPendingReference.run(...);                             // deferred, stays NULL
    }
}

The deferred path is resolved by translateMemoryReferences, whose only call site is inside applyMirrorPage (:1913):

if (page.domain === "memories") {
    translateMemoryReferences(statements);
    repairNullClobberedMemoryRows(statements);
}

So pending translations require the module to keep pushing changefeed pages. Drop authority first and they are orphaned.

repairNullClobberedMemoryRows (:1663) exists and is clearly aimed at this class of damage, but it only repairs source_type and importance:

if (sourceType === null && importance === null) continue;
statements.repairMemory.run(sourceType, importance, candidate.id);

superseded_by_memory_id is not covered, even though mirror_live_memory_rows.full_row_snapshot carries it (110 of 449 snapshots on this store hold a non-null value) — so the data needed for the repair is already on hand.

Reproduction

1. Long-lived store, memories with superseded_by_memory_id set by ordinary merge/curate activity
2. Enable transform_mode: "rust"; authority prepare succeeds and the mirror begins projecting
3. While mirror_pending_references is non-empty, revert to transform_mode: "ts" and restart
4. Pending references are cleared; affected pointers remain NULL permanently

Measured here:

mirror_pending_references   53 mid-run  →  0 after
superseded pointers         706 → 643        (63 nulled)
  of those, target resolvable  61            ← genuine metadata loss
  of those, dead links          2            ← would be dropped anyway
updated_at bumped             0             ← silent
other columns diverged        0             (content/status/category/merged_from all identical)

Detection (against a pre-activation backup):

SELECT COUNT(*) FROM memories n JOIN old.memories o ON o.id = n.id
 WHERE o.superseded_by_memory_id IS NOT NULL
   AND n.superseded_by_memory_id IS NULL;

Why 61 at once: mirror_identity covered 263 of 3101 memories, so a supersede target frequently isn't projected yet when its source row is applied. Those all take the deferred branch simultaneously.

Impact

• Silent, unaudited loss of merge-history metadata
• Recoverable only from a backup predating activation
• Larger stores are hit harder — more cross-batch targets miss on first resolve
• Made worse by the fact that reverting rust mode is the natural operator response to any rust-mode problem, which is exactly the action that makes the loss permanent

Not caused by the seeding change in #378: that strips pointers only from the seed payload and would account for 2 rows here; 63 were nulled, all with updated_at untouched, all inside the mirror's coverage. Independent of #377/#378.

Suggested fixes

Any one of these closes it; the first is the smallest.

1. Extend repairNullClobberedMemoryRows to cover superseded_by_memory_id. The snapshot already carries it, and the function already runs under a dirty flag. Resolve the snapshot value through mirror_identity and restore, skipping targets that don't resolve.
2. Drain pending references on deactivation. When authority is released, translate what can be translated and restore the rest from mirror_live_memory_rows before clearing.
3. Don't null first. Leave the existing value in place and overwrite only once the translation resolves, so an interrupted cycle degrades to a stale pointer rather than a lost one.

(3) is the most robust — it removes the window entirely — but is the largest change. (1) is a contained fix that reuses machinery already present.

Happy to send a PR for whichever you prefer.

Environment

• fork on upstream master; crates/ byte-identical to upstream, both fork features TypeScript-only
• subc daemon 0.10.0, module healthy throughout (health ok, consecutive_error_count 0, 32ms dispatches)
• rust mode otherwise worked in this cycle: decision=HARD served_from=transform applied=true, materializations non-zero
• data restored from backup via withPrivilegedWriter; post-restore diff vs backup is exactly the 2 dead links, zero drift elsewhere

Related: #375/#376 (config threading), #377/#378 (seed-set supersede pointers).

[magic-alfonso]

Confirmed and fixed. Pending supersede translation was only driven by applyMirrorPage, so a
completed authority drain could return TypeScript ownership with a resolvable pending reference
still queued. The drain now performs one final translation pass before ownership returns; references
whose target still lacks an identity remain durably recorded in mirror_pending_references rather
than being silently discarded. The regression covers both outcomes, and removing the final
translation call makes the pointer assertion fail.