feat(proof): stub-win sim scores against sealed baseline

[echobt]

Summary

Option A (this PR, lasting): under PROOF_FORCE_SIM (eval_backend=sim), a sealed topic scores with harness numbers relative to the sealed vector (sim_win_document):

• holdout NLL ≤ baseline + quality floor
• each scored split ≤ baseline + epsilon_topic_max_regress
• tokens_per_sec ≥ ref × (1 + epsilon_rel) when that is the primary

Skill-only sim_document stays NLL ≥ 1.0 and cannot beat a real ~0.29 seal. StubScorer::win skill=0.95 also fails that floor. The Lium path never uses this helper. PROOF_SIM_STUB_WIN is a leftover no-op — no extra host env.

Option B (Développeur, paused): reseal staging baselines to BASELINE_SKILL=0.40 (NLL ≈ 2.94), resign topics, retest submit→awaiting_admin on staging only. Paused — Mathis redirected that lane to prod RLM E2E (1× GPU). This lane does not reseal or edit staging host files.

Not production. No set_weights. No Lium rent. No secrets. Do not merge as part of the E2E window.

Live staging 2026-09-07 (pre-A binary)

Origin: http://159.223.159.205/challenge/proof (eval_backend=sim, force_sim=true, can_score=true, baseline_sealed=true, offer openrouter-glm53flash-v0 open). staging.api.joinbase.ai still answers a stale Lium / fail-closed instance — do not POST there.

topic	HTTP	id	state	notes
(empty)	400	—	—	topic_id is required
not-a-real-topic	400	—	—	unknown topic
dt-no-ib-v0	201	pf_0000000000000002	rejected	QualityFloor 2.827 vs baseline 0.291; ThroughputMiss 113.9 vs 213.4
muon-vs-adamw-10m-v0	201	pf_0000000000000003	rejected	NllMiss 3.042 vs baseline 0.344

Receipts: "provider":"sim". After this PR is deployed on that host (force_sim already on), --probe should reach awaiting_admin without a reseal.

Tests

• cargo test -p proof-eval stub_win — skill=0.95 and skill=1.0 still NLL≥1.0 vs 0.29 seal; sim_win_document meets the three inequalities and passes judge; Lium ignores the helper
• cargo test -p proof-eval sim_plus_sealed_uses_relative_harness — eval_after_freeze Sim+sealed uses the relative harness
• cargo test -p proof-http sim_stub_win_submit_reaches_awaiting_admin — tight 0.29 seal → awaiting_admin / eligible=true under Sim
• cargo test -p proof-http --lib — submit contract + fail-closed 400/503
• cargo test -p proof-challenge-bin --test submit_e2e — --force-sim binary, both topic ids

Runbook: docs/runbooks/proof-submit-e2e.md

Greptile

Every PR is reviewed by Greptile before merge. Config: .greptile/.

• Greptile has reviewed this PR; findings are fixed or answered
• If the bot was silent, I commented @greptileai review

Test plan

• cargo test -p proof-eval --lib
• cargo test -p proof-http --lib
• cargo test -p proof-challenge-bin --test submit_e2e
• Live staging submit→score on the current host binary (201 rejected vs seal; A not deployed yet)
• cargo clippy on touched crates
• Full cargo test --workspace not run

Risk

Staging/dev only. Live Lium scoring unchanged. Droplet compose overlays stay sim-off.

Naming

I did not rename BASE_* environment variables, deployed host paths, GHCR package names, or base-*-v1 domain tags.

  

[greptile-apps]

Greptile Summary

This change adds forced-Sim proof evaluation, submission validation coverage, and an operational submission probe. Two reproduced safety issues in deploy/scripts/proof-submit-e2e.sh must be addressed before merging: the probe can be directed to an unapproved production alias, and its shared temporary response files can be redirected through local symlinks.

Confidence Score: 1/5

Not safe to merge until the operational probe restricts its destinations to approved environments and stops using predictable shared temporary files.

Both reported failures were reproduced with the actual script against isolated loopback services, including observed mutating requests and observed symlink write-through.

Files Needing Attention: deploy/scripts/proof-submit-e2e.sh needs an approved-origin allowlist and private, cleaned-up response storage.

Security Review

The operational probe can issue mutating submission requests when a production service is reached through an IP address or unlisted DNS alias. It also writes response data through attacker-precreated symlinks in shared /tmp, allowing a local user to overwrite files writable by the operator.

T-Rex Logs

What T-Rex did

• T-Rex produced a proof for a posted P1 finding and referenced the corresponding review comment for details.
• T-Rex produced a second proof for another posted P1 finding, with multiple supporting logs attached.
• T-Rex produced a third proof for a posted P1 finding with no artifacts attached.
• T-Rex validated the proof-submit guard harness by running the harness script and reviewing the results, noting hostname rejection and successful local IP-literal POSTs.
• T-Rex completed end-to-end validation of proof submission and confirmed the write-through behavior via symlinks to response payloads.

_{Ran code and verified through T-Rex}

Comments Outside Diff (2)

1. General comment

   Production-write guard can be bypassed with a production IP address or unlisted DNS alias

   • Bug
     • refuse_prod only searches the supplied URL string for two hostname substrings. A caller can supply a production service IP address or an alternate DNS alias that does not contain network.cortex.foundation or chain.joinbase.ai; the guard then permits the probe and its POST requests. The local harness demonstrated identical allowed execution using http://127.0.0.1:39101, including POSTs to /v1/submissions, while the literal denied hostname was rejected before any request.
   • Cause
     • deploy/scripts/proof-submit-e2e.sh:30-35 uses echo "$url" | grep -Eq "$PROD_HOSTS" rather than parsing and canonicalizing the URL host, resolving it, and enforcing an allowlist of approved local/staging origins.
   • Fix
     • Parse the URL authority and fail closed. Prefer an explicit allowlist of approved staging/local origins; if IP destinations must be supported, resolve approved names and reject production IP ranges/addresses after DNS resolution. Revalidate each actual connection target, and retain redirects disabled (or validate every redirect hop before allowing a request).

   _{Ran code and verified through T-Rex}

2. General comment

   Predictable curl response paths permit symlink write-through

   • Bug
     • An attacker able to create files in shared /tmp before an operator runs this script can replace any of the three predictable response paths with a symlink. The operator's curl invocation follows that link and truncates/writes the target with the response body. The demonstrated local run altered all three symlink targets.
   • Cause
     • The script uses fixed /tmp filenames with curl -o instead of creating private unique files, and performs no symlink-safe file creation or ownership validation.
   • Fix
     • Create a private temporary directory with tmpdir=$(mktemp -d) and trap 'rm -rf -- "$tmpdir"' EXIT; place each response output in that directory (for example, "$tmpdir/proof-empty.json"). Do not use predictable shared /tmp leaf filenames.

   _{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "test(proof): lock sealed-relative sim in..." | Re-trigger Greptile}

[greptile-apps]

Bypassable Production Protection

If a production service is addressed through its IP address or an alternate DNS name, this guard allows the probe because it only rejects URL strings containing two literal hostnames. The script then sends mutating submission POST requests to that origin, so an operator can unintentionally write test submissions to production. Require an explicit allowlist of approved local or staging origins and fail closed for every other destination. This must be addressed before merging.

How this was verified: A loopback IP URL passed the guard and received both submission POSTs, while a URL containing the denied hostname was rejected.

Knowledge Base Used:

• Deployment, runtime operations, and image updates
• Deployment automation and infrastructure

Artifacts

Evidence from the check

• Authored executable harness that starts only loopback mock services and invokes the supplied script against denied-host, IP-literal, and redirect cases, ending with assertions about observed requests.

Command output from the check

• Captured command output showing the implemented substring denylist and the guarded proof POST call sites before runtime validation, establishing the code under test.

Command output from the check

• Captured local harness execution showing hostname rejection, IP-literal mutation requests to the loopback mock, and no request to the redirect target, confirming the alias/IP bypass but disproving redirect following.

_{Ran code and verified through T-Rex}

[greptile-apps]

Shared Temporary Files

The probe writes responses to predictable filenames in shared /tmp locations at this call site and the equivalent calls later in the script. A local user can pre-create one as a symlink; curl -o follows it and overwrites the linked operator-writable file when the script runs. Create a private directory with mktemp -d, store all responses there, and remove it with an exit trap. This must be addressed before merging.

How this was verified: Pre-created symlinks for all three response paths remained symlinks and each linked target was overwritten by the local response body.

Knowledge Base Used: Deployment automation and infrastructure

Artifacts

Evidence from the check

• The authored executable starts only loopback HTTP servers, runs the supplied script in before and symlink-precreated modes, and inspects targets; it is the executed source.

Command output from the check

• The command capture records the complete local harness source that was executed, including its loopback-only server setup and symlink assertions; it provides reproducible test input.

Command output from the check

• The loopback-only before run completed successfully and created ordinary response files at the three script-selected paths; it establishes normal behavior.

Command output from the check

• The loopback-only after run completed successfully while all three response paths remained symlinks and every linked target was overwritten with the response body; this confirms the vulnerability.

Command output from the check

• The captured source search identifies all three fixed curl output paths and finds no mktemp, trap, or removal command in the script; it confirms affected locations and missing cleanup.

_{Ran code and verified through T-Rex}