feat(proof): stub-win sim relative to sealed baseline (reland #232)

[echobt]

Summary

Reland of squash-merged #232 (c5c581a) onto current main (be0529f7 / #234). That squash landed on a stale parent (1ffd2599, which also carried unmerged journal/atlas work) and never reached main, so staging still scores with absolute sim_document (NLL ≥ 1.0) against a sealed ~0.29 baseline and rejects.

Option A (this PR): under PROOF_FORCE_SIM (eval_backend=sim), a sealed topic scores with harness numbers relative to the sealed vector (sim_win_document):

• holdout NLL ≤ baseline + quality floor
• each scored split ≤ baseline + epsilon_topic_max_regress
• tokens_per_sec ≥ ref × (1 + epsilon_rel) when that is the primary

Skill-only sim_document stays NLL ≥ 1.0 and cannot beat a real ~0.29 seal. StubScorer::win skill=0.95 also fails that floor. The Lium path never uses this helper. PROOF_SIM_STUB_WIN is a leftover no-op — no extra host env.

Not production. No set_weights. No Lium rent. No invented digests. Droplet overlays stay sim-off (assert-compose-matrix.sh also refuses PROOF_SIM_STUB_WIN on staging/prod).

Conflict resolutions vs #233 / #234

• fix(proof-harvest): log stdout_tail on PROOF_EVAL_OK refuse #233 (stdout_tail in proof-harvest) did not overlap StubWin files — kept as-is.
• fix(proof): stage proxy model + holdout for live score #234 added EvalError::ProxyModelMissing / HoldoutStoreMissing in proof-eval plus live-asset notes (PROOF_PROXY_MODEL_DIR, PROOF_HOLDOUT_STORE). Kept those, then applied sim_win_document and the sealed argument to eval_after_freeze.
• docs/runbooks/staging-testnet-e2e.md: kept fix(proof): stage proxy model + holdout for live score #234’s proxy/holdout 503 note and feat(proof): stub-win sim scores against sealed baseline #232’s sim --probe contract / droplet IP.
• Cargo.lock: restored from main, then incrementally added only the new test deps (crypto, hex, reqwest, sha2 on proof-challenge-bin; reqwest on proof-http).

Tests (local + CI)

• cargo test -p proof-eval --lib — 11 passed including stub_win_* and sim_plus_sealed_uses_relative_harness
• cargo test -p proof-http --lib — 19 passed including sim_stub_win_submit_reaches_awaiting_admin
• cargo test -p proof-challenge-bin --test submit_e2e — force_sim_binary_scores_staging_topic_ids passed
• cargo fmt --all -- --check and cargo clippy on touched crates (-D warnings) passed
• GitHub ci / fmt · clippy · test · deny · xtask green on a2674f3a

Greptile

Every PR is reviewed by Greptile before merge. Config: .greptile/.

• Greptile has reviewed this PR; findings are fixed or answered
• If the bot was silent, I commented @greptileai review

Two P1s answered on-thread (intentional sim Option A; documented harness_success_rate fail-closed custom). No product change.

Test plan

• cargo test -p proof-eval --lib
• cargo test -p proof-http --lib
• cargo test -p proof-challenge-bin --test submit_e2e
• cargo fmt --all -- --check on touched files
• Clippy on touched crates
• CI fmt · clippy · test · deny · xtask green
• Full cargo test --workspace not run locally (CI ran it)

Risk

Staging/dev only. Live Lium scoring unchanged. Droplet compose overlays stay sim-off. Does not touch production pins or harvest.

Naming

I did not rename BASE_* environment variables, deployed host paths
(/opt/base, /run/base, …), GHCR baseintelligence/base package names, or
base-*-v1 cryptographic domain tags, unless this PR’s purpose is a coordinated
cutover documented in docs/NAMING.md.

  

[greptile-apps]

Greptile Summary

Summary

• Forced simulated evaluation can turn arbitrary sealed-baseline submissions into eligible passing records without evaluating their artifacts, and those records enter payout inputs.
• Simulated scoring for the supported custom metric cannot pass when a valid sealed baseline has no custom value.

These failures must be resolved before merging.

Confidence Score: 3/5

Not safe to merge: simulated results can affect payout inputs without artifact evaluation, and a supported custom scoring path cannot succeed under valid sealed-baseline data.

Two independently reproduced failures affect the proof evaluation and scoring flow.

Files Needing Attention: crates/proof-eval/src/lib.rs needs changes to prevent simulated wins from becoming payout-eligible and to handle required custom measurements consistently.

T-Rex Logs

What T-Rex did

• T-Rex posted a P1 finding proof with the Live Lium control test output, the forced-sim sealed-baseline HTTP submission output, and the executed forced-sim reproduction test source.
• T-Rex posted a second P1 finding proof with the existing simulation custom-value test output, the authored focused Rust reproduction test source, and the focused sealed custom baseline scoring reproduction output.
• Two additional P1 findings were posted.
• General-contract validation confirmed the pre/post-forced-sim behavior, including the status and submission outcomes observed under the forced-sim path, and the repro source is retained as an artifact.
• Focused contract validation showed harness evaluation reporting a missing custom_value and that no HTTP endpoint was exercised, with the repro source preserved.

_{Ran code and verified through T-Rex}

Comments Outside Diff (2)

1. General comment

   Forced simulation fabricates sealed-baseline wins that enter payout inputs

   • Bug
     • In crates/proof-eval/src/lib.rs:681-688, the changed EvalBackend::Sim branch selects sim_win_document whenever the HTTP handler supplies a sealed baseline. That helper at crates/proof-eval/src/lib.rs:569-639 unconditionally marks the agent clean/reproduced and constructs harness metrics relative to the sealed baseline so they satisfy the relevant NLL, throughput, latency, or custom-metric win condition. It neither loads nor evaluates the submitted artifact. The HTTP handler at crates/proof-http/src/lib.rs:308-343 always supplies Some(&sealed), then persist_scored at lines 441-507 marks a passing result AwaitingAdmin and immediately records a passing MinerTopicRun. proof_challenge::store_runs and emission_scores (crates/proof-challenge/src/lib.rs:93-103,54-59) use these recorded runs for payout_lattices without filtering on submission state. The runtime repro demonstrated the consequence for dt-no-ib-v0 and muon-vs-adamw-10m-v0: both received 201 Created / awaiting_admin / eligible true and persisted 200 OK passing verdicts, while status showed no live harvest was wired.
   • Cause
     • The new sealed-baseline argument changes sim from a deterministic non-winning fallback into sim_win_document, which derives favorable values from the reference baseline and hard-codes a clean reproduced verdict. The persistence path treats this result exactly like an evaluated pass and updates the payout input before any administrator action.
   • Fix
     • Do not permit EvalBackend::Sim results to become eligible submissions or payout runs. Restrict forced sim to explicit test-only builds/environments, or persist sim submissions as non-eligible diagnostics. Independently, make payout collection require a verified live (lium) receipt and/or an accepted champion state rather than accepting every recorded passing run.

   _{Ran code and verified through T-Rex}

2. General comment

   Supported custom topics can be opened but simulated sealed-baseline scoring always rejects missing custom evidence

   • Bug
     • A supported custom topic (harness_success_rate) passes TopicDocument::validate and can be open with a sealed recipe/metrics commitment whose measured custom_value is absent. On the simulated sealed-baseline path, eval_after_freeze returns successfully but the resulting harness retains custom_value: None; judge_topic then rejects it with GateFail::EvidenceMissing { field: "custom_value" }. This affects all simulated submissions for such a topic. Security impact: none identified; this is an availability/correctness failure for the advertised custom-topic path.
   • Cause
     • supported_custom() advertises harness_success_rate (crates/proof-eval/src/lib.rs:132-141), while topic validation accepts a sealed custom baseline without requiring a measured custom value (crates/proof-task/src/topic.rs:665-685, 853-855). sim_win_document maps the sealed optional value unchanged (crates/proof-eval/src/lib.rs:606-608) and writes it to the harness (line 637). The scoring gate requires both values and fail-closes on either absence (crates/proof-score/src/lib.rs:385-397). Lines 662-664 only call offer.serves_topic and are not causal.
   • Fix
     • Do not advertise or permit opening harness_success_rate topics until its measurement sidecar supplies a real custom metric and baseline value. Alternatively, require BaselineMeasurement.custom_value: Some(finite) for custom topics and make the simulated evaluator produce an explicitly test-only measured custom value only if that is a valid simulation contract.

   _{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "feat(proof): stub-win sim relative to se..." | Re-trigger Greptile}