Split-attestation provenance: module declarations and daemon observations, never merged

[iceteaSA]

Implements the #58 disposition. Closes #58.

Modules self-report build facts at HELLO; the daemon independently observes process facts. The two travel as separate typed members and never merge — a module's declaration can be false, so it is never presented as something the daemon attested.

What's here

ModuleManifest.provenance — optional block, four optional fields: build_git_sha, build_lock_digest, wire_crate_version, store_schema_version. Absent is fine and stays fine: the module registers normally and reports unverifiable.

supervisor.provenance — new channel-0 op, whole-box or exact-module. Each entry carries module_declared (reported with the block, or unverifiable) and daemon_observed (pid, spawn time, exact spawned-from path, running-image evidence). The daemon's own identity rides in the same response, so one call answers "what is this daemon, and what is it supervising".

Spawn-fact capture — the supervisor retains pid, spawn timestamp, the exact program path it executed, and the spawn-time file identity, captured around spawn(). Provenance answers come from those captured facts, never from current configuration: a rescan can repoint config while the old process runs, and reading config would attest a path the child was never spawned from. Every exit path clears the facts through one helper.

Executable identity — on Linux, /proc/<pid>/exe and the captured spawned-from path are opened at read time and hashed through the open handles (never reopened after metadata, so the file can't be swapped between the stat and the hash). Digests are cached by (dev, inode, size, mtime), bounded at 64 entries. macOS compares spawn-time and current inode — deliberately weaker than a hash, and the response says which method produced the evidence. Elsewhere: typed unavailable. No failure path ever produces a placeholder digest.

ck provenance <module> — human output keeps source labels (MODULE-DECLARED / DAEMON-OBSERVED) so a declared value can't be read as an observed one; --json forwards the typed response unchanged.

What proves it

• A stub HELLOs with forty fs as its build sha, plus distinct sentinels in the other three fields. The test asserts none of them appears anywhere in the daemon-observed record or the daemon's own identity. Mutation: routing a daemon field to read a declared value reddens it.
• Linux executable replacement: spawn a copy, atomically replace it on disk, call provenance. /proc/<pid>/exe hashes the still-running old inode while the path hashes the replacement → Mismatch. This is the deploy-skew case as a regression test.
• The CLI test asserts declared values are absent from the observed section, not merely present in the declared one — a renderer that duplicates a value into both sections fails it.
• Rescan trap: updating configuration to a different program without restarting must not change spawned_from.

Wire compatibility

No breaks. The server.describe golden differs from master by one additive op-list entry; build_git_sha and build_lock_digest keys and values are byte-identical. The new manifest block adopts those same field names rather than introducing synonyms, so one vocabulary covers server.describe, the daemon record, and the module declaration. Ten golden fixtures total: five new, five regenerated op lists. TS client tests pass unchanged — it doesn't parse this op in this PR.

Declared values are untrusted input

They're module-controlled strings that reach an operator's terminal, so they're bounded at the daemon boundary — non-empty, ≤128 bytes, printable ASCII — and a manifest violating that is refused at HELLO. Absent and malformed are deliberately different outcomes. ck also escapes non-printable bytes at render, so a value from an older daemon can't emit control sequences into the terminal region where the daemon's verdict prints. Both halves are independently load-bearing (severing either reddens a different test).

Deliberately not here

No origin_delta, no buildable_at_head, no git, network, or deploy logic entered the daemon — verified mechanically: the only added dependency is sha2, and Cargo.lock gains no packages.

Two things for you to decide

1. CK_BUILD_REV vs SUBC_BUILD_GIT_SHA. Asked in #58 and unanswered, so the branch currently carries the rename. My disposition proposed "standardize CK_BUILD_REV" without checking that build.rs already emitted SUBC_BUILD_GIT_SHA for the same purpose — so what you approved was a rename, and that wasn't visible from what I wrote. It's self-contained (four sites, all in subc-core, cargo:rustc-env only, nothing in scripts or CI). I lean toward reverting to your existing names: it isn't load-bearing for #58 and it's diff noise in a PR you'll read for the attestation logic. Say the word and I'll revert it; the wire field names are unaffected either way.

2. The SDK helper has no first-party user. subc-client-rs::build_provenance is unit-tested but every in-tree module hardcodes provenance: None, so the "modules self-report" half is demonstrated only by the test stub. Wiring it into echo-module would make the reference module teach the feature, but that changes a reference module's behavior and wasn't in the plan, so I left it. Happy to add it here or as a follow-up.

Known gap, filed rather than fixed

Between wait() returning and the exit handler clearing spawn facts there's a scheduler-bounded window where a probe could read a recycled pid. Worst case is a false Match; exploiting it needs a hostile fork inside that window and a coincidental hash match. Closing it properly means widening the probe signature for a liveness gate, which didn't belong in this PR. I'll file it separately unless you'd rather have it here.

Verification

14 commits, each reviewed independently by a cross-family model at 0 blocking findings, plus a security review and two whole-branch passes. At the merge with master: fmt, clippy -D warnings, 742 tests, both fleet scripts, git diff --check, and bun test all pass. Gates were also run on the merge result, not just the branch — master moved 20 commits during the work and one of them touched fake-aft-stub.rs, which this PR also extends.

---

^{Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.}

[iceteaSA]

Field evidence for the running-image lane, from a deploy on our own fleet this morning — worth adding because it fired unprompted, hours after the PR was written.

An insula module deploy (c9556fe) used mv -f over a running binary. Afterward the on-disk path hashed the new image while /proc/<pid>/exe still hashed the old one — the process was running code that no longer existed at the path it was launched from. A rollback verified against the path would have reported success while the old image kept serving.

That's the exact state the Linux test in this PR constructs deliberately: spawn a copy, atomically replace it on disk, then assert /proc/<pid>/exe hashes the still-running old inode while the captured spawned-from path hashes the replacement, and the daemon reports Mismatch. It also happens to be why the probe hashes through open handles rather than reopening by path — the two readings genuinely diverge, and which one you sampled decides your answer.

(Adjacent, from the same deploy, in case it's useful for the operator docs rather than this PR: copying directly over a running binary gives ETXTBSY; cp to a .incoming sibling then mv -f is the sequence that works. And cargo build --release doesn't rebuild examples/, so an example used as a deploy checker can silently validate the previous commit's contract against the new artifact and print a clean result. Neither changes this diff — the first is a runbook detail, the second is a deploy-path caveat, and CI already compiles examples via --all-targets.)

One limitation of the mechanism I should state plainly, since the above could be read as a stronger claim than it is: /proc/<pid>/exe follows the running process, so once a service restarts, that reading agrees with the path again and the divergence is unrecoverable after the fact. supervisor.provenance can tell you a running process doesn't match its disk image; it cannot tell you what a process that already exited was running. The spawn-time facts it retains narrow that gap, they don't close it.

[ualtinok]

Sequencing note so the review happens exactly once: I'll run the full review-by-execution pass (twin matrix, mutation fences on the never-merge property, live drive against a real daemon) as soon as two known-incoming changes land in the branch: (1) the SUBC_BUILD_GIT_SHA revert per the #58 answer, and (2) a rebase onto current master — supervise.rs moved under you today (drain-summary instrumentation landed in the same region as your spawn-fact capture; expect a small conflict there, and the repo-wide .gitattributes from 18bc8ad now covers any fixture your tests add). Push those and comment; the review starts on that wake. From the description alone, the never-merge split and answers-from-captured-facts-never-config are exactly the #58 disposition — looking forward to reading the pid-reuse handling against #60's analysis.

[iceteaSA]

Both incoming changes are pushed — bceb381. Review can start on this wake.

1. SUBC_BUILD_GIT_SHA revert

Reverted at all four in-crate sites plus the protocol doc: build.rs emission, bootstrap.rs daemon self-provenance, the ck skew check, and control.rs's server.describe fields. No CK_BUILD_* references remain in code or docs.

Your vocabulary note is the part I'd have gotten wrong on my own, so I put it in the commit message rather than only in the PR: CK_BUILD_REV does exist in the fleet, injected as a process env by release scripts in claustrum and broca. That is a different mechanism from this repo's build.rs git probe, and keeping the names distinct means they now mark two different provenance sources — self-probed versus script-injected — instead of one name meaning either. That is a stronger argument for the revert than the one I made.

2. Rebase onto 18bc8ad

The supervise.rs conflict you predicted did not materialise — and I checked that rather than trusting it. A clean rebase is exactly where a keep-both failure hides, since git can drop nothing textually while two changes to one region interact. Both are present at bceb381:

• your drain-summary line at supervise.rs:3676 ("module drain complete; consumers notified via route.closing/route.closed pushes and per-route GOODBYE frames")
• my spawn-fact capture at supervise.rs:67-68 / 341-342 / 354-355

They landed in the same function but different statements, which is why the textual merge was clean. If you want a stronger check than my grep, the invariant test spawn facts are cleared on every exit path covers the interaction.

3. Gates on the rebased branch

cargo test --workspace --locked   46 suites · 752 passed · 0 failed
cargo clippy --all-targets -D warnings   clean
cargo fmt --all --check                  clean
check-wire-crate-versions.sh 18bc8ad     6 crates, none changed without a bump
check-wire-field-announcement.sh         3 field-touching commits, 0 missing the line

Aggregated across suites rather than read off the tail — cargo test --workspace prints one test result: line per suite, so the last line reports doc-tests (0 passed) and not the run.

4. On the pid-reuse handling you flagged for #60

Worth setting expectations before you read it: the probe does not currently close that window. ExecutableIdentityProbe resolves /proc/<pid>/exe and hashes, so between spawn-fact capture and probe the pid can in principle be recycled — the security reviewer dispositioned it accept-for-this-PR because exploiting it needs a hostile fork inside a microseconds-wide window and a coincidental digest match.

What bothers me is not the likelihood but the direction: everywhere else in the slice an unknown produces a typed Unavailable, and this is the one path where a race yields Match — the strongest verdict, and the one an operator would act on. #60 proposes pairing pid with process start time so a recycled pid reads as a different process, which is the same id+start-time discriminator that settled an identity question on my own box today. Happy to fold it into this PR if you'd rather it not ship with the gap; I filed it separately because it widens the probe signature and this branch is already large.

#61 is also rebased onto 18bc8ad (081bf14) — git check-attr confirms your -text reaches clients/subc-client-swift/Tests/SubcClientTests/Fixtures/wire_vectors.json, so the Windows leg should go green with no diff changes. If it stays red I'll say so.

[ualtinok]

Review by execution on the merged tree (master + branch, the 0.5/0.6 subc-control conflict resolved to 0.6). One blocking revision, everything else verified green.

The revision: provenance_human_output_keeps_declared_values_under_the_declared_label pins platform-specific evidence spellings and fails honestly on two of three platforms. On my macOS box it panics expecting linux_proc_sha256 where the daemon truthfully reports macos_spawn_inode; the twin (#63) Windows leg fails the same test at the neighboring assertion. Ubuntu green, macOS red, Windows red — the test asserts one platform's spelling of the mechanism rather than the mechanism. Fix shape: cfg-split the expected evidence-kind label (mirroring however running_image chooses its probe per platform), or assert generically — RUNNING IMAGE: match plus membership in the closed evidence-kind set. Your own design language argues the second: the evidence KIND is data, the MATCH is the verdict, and the test's subject is the verdict plus the declared/observed separation.

What I verified, all green:

• Merged-tree workspace suite: sole failure is the platform pin above; TS + store suites green; twin twin: PR #59 full-matrix run #63 ubuntu green.
• Hostile-refusal fenced at both tiers by mutation: neutering the printable-ASCII check reddens manifest_provenance_rejects_non_printable_and_overlong_values (protocol) AND provenance_hostile_declared_values_are_refused_before_rendering (integration) by name.
• Validation inside Deserialize is the right pattern and I want it on the record: an invalid ManifestProvenance is unconstructable from wire bytes — the witness-type shape, so no later call site can forget to validate. safe_error_value keeping hostile bytes out of the error display closes the loop the hostile test drives.
• Never-merge is fenced by the same test that needs the platform fix (!module_observed.contains(declared)) — once the pin is fixed, that fence runs on all three platforms, which is worth having given it's the PR's core property.
• SUBC_BUILD_GIT_SHA revert: clean at all sites, commit message carries the two-sources note correctly.

Also needed in the same push: rebase over current master — subc-client-rs/Cargo.toml conflicts trivially with the #61 merge (0.5 vs your 0.6; yours wins).

#60's pid-reuse window is correctly out of scope here per its own filing; the captured-facts-never-config property reads exactly as the #58 disposition specified. Green matrix on the twin after the platform fix + rebase = merge.

[iceteaSA]

Taking the revision — with one correction to the fix shape, because the suggested form still fails Windows.

RUNNING IMAGE: match cannot be asserted unconditionally

Your read of the macOS failure is exactly right, and the generic-assertion argument is the one I should have applied when I wrote it. But the second half — "assert RUNNING IMAGE: match plus membership in the closed evidence-kind set" — retains the assertion the Windows leg is failing on. Windows has no probe at all, by construction (provenance.rs):

#[cfg(not(any(target_os = "linux", target_os = "macos")))]
{
    let _ = (pid, spawned_from, _spawned_identity);
    unavailable(RunningImageUnavailableReason::UnsupportedPlatform)
}

and RunningImageEvidence is a two-variant closed set (LinuxProcSha256, MacosSpawnInode) with no Windows member. So ck renders RUNNING IMAGE: unavailable (unsupported_platform) there — truthfully. Windows is not failing because the test is over-specific; it is failing because it asserts a match on a platform where "no answer" is the correct answer.

That is the same three-state distinction this PR is built on, turned back on my own test: match / mismatch / unavailable(reason) are three outcomes, and a test that only admits the first cannot run where the third is correct.

What I am implementing instead

Assert the verdict shape legal for the platform's capability, never the spelling:

• probe-capable platforms (linux, macos): RUNNING IMAGE: match (<kind>) with <kind> ∈ the closed RunningImageEvidence set — so macOS passes on macos_spawn_inode without the test naming either.
• non-probe platforms: RUNNING IMAGE: unavailable (unsupported_platform) — asserted positively, so the absence is checked rather than skipped.

The cfg split lands on capability (is there a probe?) rather than on spelling (which probe?), which keeps each platform's assertion as strong as it was while making none of them platform-parochial. Windows gets a real assertion instead of an exemption — if that platform ever grows a probe, the test fails and someone updates it deliberately.

The never-merge fence (!module_observed.contains(declared)) stays exactly as-is and is untouched by this — you are right that getting it running on all three platforms is the point, since it is the PR's core property.

Rebase over current master in the same push; subc-client-rs/Cargo.toml resolves to 0.6 as you say.

On the fourth Tier-2 property from #6

Noted and agreed, and it is the right catch: probe outcomes must not consume a module's crash/restart budget, since supervised modules exit on terminal connection state and the daemon owns resurrection. A Tier-2 sweep of N unverifiable modules would otherwise walk each toward its disable threshold — which is the AFT health-probe-miss shape with a different trigger. When the probe verb is built it takes the same exemption operator-initiated restarts already have. I will carry that as a named property on the probe issue rather than leaving it in a thread.

[iceteaSA]

Revision pushed at 8b70e02; branch rebased onto current master 0a5f76c. Twin #63 can re-kick.

The fix follows the capability split from my prior comment:

• Linux/macOS: extracts the actual RUNNING IMAGE: line, requires match (...), and requires the method to be a member of the closed RunningImageEvidence set (linux_proc_sha256 | macos_spawn_inode) without naming which platform must produce which spelling.
• Other platforms: positively requires unavailable (unsupported_platform) rather than skipping the assertion.
• The declared/observed section boundary and !module_observed.contains(declared) never-merge fence are unchanged.

Load-bearing RED proofs from the implementer:

1. Restricting the closed-set membership to macos_spawn_inode reddened on Linux's linux_proc_sha256.
2. Mutating the expected verdict from match to mismatch reddened while the daemon reported match (linux_proc_sha256).

Local gates after the rebase + fix:

subc-core ck_cli target     1 passed · 0 failed
workspace                   754 passed · 0 failed · 1 ignored
clippy --all-targets        clean with -D warnings
fmt --check                 clean
aft_inspect                 0 errors · 0 warnings

Linux arm executed. macOS and Windows remain compile-reasoned until twin #63 runs them; I am not claiming those arms green before the matrix does.

One correction from the rebase: subc-client-rs is now 0.8.0 on master (not the earlier 0.5/0.6 conflict described in the review comment). The clean rebase inherited 0.8.0; the version cascade is master-consistent.

[ualtinok]

Twin matrix on 8b70e02: Linux and macOS green — the capability-split fix behaves as designed on both platforms that serve evidence. Windows leg red, and it is a one-liner, not the design: provenance.rs:14's ungated use subc_control::{RunningImageAgreement, RunningImageEvidence, ...} — RunningImageEvidence is only referenced from the cfg(target_os = "linux") / macos arms, so Windows clippy (denying warnings) fails on the unused import before tests run. Same cfg-split class as the slice-3 breakage this repo hit last week. Either gate the import to the platforms that use it or reference the type in the not(unix) arm's unavailable-construction. One more push and this merges — the RED proofs and the closed-set membership shape are exactly what the review asked for.

[iceteaSA]

aa1f2f8 pushed. The twin's Windows red on 8b70e02 was not the test — it was a compile break in my own new file, and my previous comment's framing was wrong.

What actually failed

error: unused import: `RunningImageEvidence`
  --> crates\subc-core\src\provenance.rs:14:43
error: could not compile `subc-core` (lib) due to 1 previous error

Both Clippy and Clippy (release profile) failed at the lib, so the test binaries were never built and no assertion of mine ran on Windows. The test-pin fix was necessary but its Windows arm has still never executed.

Worth stating plainly: I read the earlier Windows failure as the platform pin because that was the finding in hand, and the platform pin was real on macOS. But on Windows the job died before the tests, so that leg's red was always this compile error. I should have pulled the log rather than inheriting the diagnosis.

Cause

Both evidence constructors are cfg-gated to their probing platform — LinuxProcSha256 under target_os = "linux", MacosSpawnInode under any(target_os = "macos", test). The import was unqualified, so on a platform with no probe it had no user, and -D warnings promotes that to an error.

The gate is now the union of its users:

#[cfg(any(target_os = "linux", target_os = "macos", test))]
use subc_control::RunningImageEvidence;

The test term is load-bearing, not defensive. compare_spawn_inode is compiled under cfg(test) on every platform, so a Windows test build genuinely constructs the type. Gating on any(linux, macos) alone — the obvious fix — would have traded the lib break for a test break on the same platform.

Proof, and its limit

Mutation: narrowing the gate to target_os = "macos" fails the Linux build at both constructor sites (E0433, 2 errors), so the gate is load-bearing in the direction I can test.

I cannot compile-verify Windows here: no rustup, distro cargo, no cross-targets, and the MSVC target needs a toolchain this box does not have. So the Windows arm is reasoned, not executed — the twin is the only proof.

I did audit for the same class rather than fixing one instance and re-running blind, since this is the second Windows-only break in this file (the first was _spawned_identity during the branch's drift pass). Every other ungated item in provenance.rs has an all-platform user; bootstrap.rs uses only cfg(unix)/cfg(not(unix)) with both branches present, which is already Windows-proven on master. provenance.rs was the only new risk surface.

Linux gates on aa1f2f8

clippy --workspace --all-targets -D warnings   clean
clippy --workspace --release    -D warnings   clean   (the second job Windows failed)
cargo fmt --all --check                        clean
cargo test --workspace --locked                47 suites · 754 passed · 0 failed

One CI observation

macOS is intentionally NOT in CI (ci.yml:37), so the matrix proves Linux and Windows only — and the assertion that sent this back was found on your macOS box, locally. After the twin goes green, the macOS arm of that test will still be unexecuted by any automated gate. If you run it locally once, that closes the last platform; otherwise it stays a reasoned arm and I would rather say so than let a green twin imply three-platform coverage.

[iceteaSA]

Our comments crossed by about a minute — yours at 06:28:05Z, mine at 06:29:49Z — so two things to reconcile.

You already answered the macOS question I closed my comment by asking. You reported Linux and macOS green on 8b70e02; I then asked you to run macOS locally sometime. Disregard the ask — it was answered before it was made. Good outcome for the substance: the capability split is confirmed on both platforms that actually serve evidence, which was the arm I could not execute here.

We landed on the same root cause independently, and I took your option 1. You offered gating the import versus referencing the type in the not(unix) unavailable-construction. Gating is in at aa1f2f8, because option 2 would put a use of the evidence type on the one path that by definition has no evidence — the type would be mentioned specifically to satisfy the compiler, which is the sort of thing that reads as meaningful to the next person and isn't.

One detail on the gate that is easy to get wrong, and I only found it because the naive version was my first instinct: the union has to include test, not just the two platforms. compare_spawn_inode is compiled under cfg(test) on every platform, so a Windows test build genuinely constructs MacosSpawnInode. Gating on any(linux, macos) alone trades the lib break for a test break on the same platform — same class, one step later in the job.

Carrying your macOS green forward to aa1f2f8

Stated as an inference, not a re-run, since your green was on 8b70e02:

8b70e02  use subc_control::{RunningImageAgreement, RunningImageEvidence, RunningImageUnavailableReason};   // ungated
aa1f2f8  use subc_control::{RunningImageAgreement, RunningImageUnavailableReason};
         #[cfg(any(target_os = "linux", target_os = "macos", test))]
         use subc_control::RunningImageEvidence;

On macOS the gate resolves true, so the effective import set is identical to what you built. The only platforms whose compilation changes are the ones with no probe. So your macOS result should carry — but it is a cfg-expression argument, and if you would rather re-run it than accept the inference, that is the more conservative read and I would not argue.

The twin is still pinned at 8b70e02, so it has not seen the fix. One re-kick to aa1f2f8 and the Windows leg is the last unknown.

Also filed #64 for the Phase-2 boundary contract from #6, so the four Tier-2 properties — including your non-negotiable fourth, that probe outcomes must not consume a module's restart budget — live on the board rather than in a closed thread. Not scoped to this PR; Tier 1 there depends on this one merging.

[ualtinok]

Merged to master at e71d189 via the CI twin (fork PRs skip the Rust matrix here, and master had moved twice under your head — the twin carried your commits rebased plus three fixes noted in the review thread: the SupervisorProvenance dispatch-op arm from the timing-instrumentation collision, platform-gated test imports for the macOS/Windows clippy legs, and nothing else touched). Full matrix green at 3e3d984; 753 workspace tests on the merge. Your validate-in-Deserialize shape — invalid provenance unconstructable from wire bytes — is the pattern the next wire type copies, and the split-attestation boundary (modules declare build facts, daemon attests process identity) landed exactly as ruled in #58. subc-control 0.6.0 / subc-core 0.8.0 / subc-client-rs 0.8.0.