feat(proof): snapshot journal payouts and listed-op flop traces

[echobt]

HOLD (Mathis FINAL, 2026-09-07)

• No master/control-plane Lium rent. Strict miner BYOK only (X-Lium-Api-Key / miner-paid pod). Do not rent on the master LIUM_API_KEY.
• Cancel check just now: GET https://lium.io/api/pods → 0 pods. Nothing to terminate. No rent was created this session. LIUM_LIVE_WRITES was never implemented.
• Stop Lium live work. Wait for runtime CI + Greptile only.
• Do not merge, deploy, or set_weights without Mathis go.

Summary

Finish the missing payout-journal integration on checkpoint/proof-production-readiness-20260907 and add independently recomputed compute traces. Proof/Bounty stay 8000/2000 bps. No merge to main, no production deploy, no on-chain set_weights.

Payout / durability

• emit_epoch_from_store calls store.snapshot_durable().await? on every emission and propagates journal errors.
• store_runs / store_scores return Result and no longer swallow failures into empty maps. Journaled sync readers still refuse.
• bins/proof-challenge reloads via list_durable(); a failed snapshot is a boot error, not a silent empty store.
• Migration 0031 on this checkpoint already creates proof_submission_id_seq. Do not replay 0031 on an already-migrated env.
• New 0032 unique index on frozen submission_digest makes identical retries idempotent. Concurrent proof_topic_run updates remain last-writer-wins. Network loss during COMMIT is still untested.

Reproduction / FLOPs

• Holdout measurement records a listed-op compute trace (mm / addmm / bmm / math SDPA). Unlisted compute-shaped ops refuse rather than undercount.
• Controller-side twin: proof_research::flop_total recomputes the same total from the retained op list. A supplied flops_used is never trusted.
• proof-eval score|baseline still refuses success without PROOF_TRAINING_EVIDENCE_FILE carrying a matching retained trace. eval/baselines/adamw.py is a parameter lock, not executable training.
• artifact_fingerprint now hashes file bytes (sorted name + SHA-256), not the directory path.
• This is not a claim that every training recipe was reproduced. A one-matmul fixture does not attest every recipe. Journal rows are JSON, not artifact blobs (proof-measure 64 MiB tar cap; proof-research 16 MiB / 1 MiB retained-artifact caps).

The v1 binary still has no automatic emission loop. can_score is not payment.

Greptile

• Greptile has reviewed this PR; findings are fixed or answered
• If the bot was silent, I commented @greptileai review

Test plan

Targeted executables only (not a full workspace re-run). Disposable Postgres: labeled cortex_proof_payout_ed26 on localhost:55432, isolated base_test_* schemas via db::test_pool. SQLX_OFFLINE=true for compile.

• proof-challenge lib: 6 passed, 1 ignored (journal path then run with --ignored: pass)
• proof-store lib: 4 passed
• proof-store --test durability -- --ignored: pass (incl. digest retry)
• proof-http --lib committed_terminal -- --ignored: pass
• proof-research --lib flops: 3 passed
• cargo fmt --all -- --check
• cargo clippy -p proof-challenge -p proof-store -p proof-research -p proof-challenge-bin --all-targets -- -D warnings
• Python test_contract.py + test_compute_trace.py (except torch collector): 34 passed, 1 deselected
• data-only harness 14 CPU tests: not re-run here (no torch in this environment)
• cargo test --workspace: not re-run

Connected (session secrets via /tmp env-file; never committed):

• Astra judge acknowledgement: has_astra=true, astra_models_count=27, astra_judge=ok (synthetic ack, not science)
• Lium --preflight earlier: GET-only, found RTX 4090 at 0.32 USD/h, refused rent (live_writes_enabled=false, exit 2).
• Owner cancel (now): GET /pods on the master key → 0. No DELETE needed. Live Lium work stopped. Miner BYOK only.

Risk

Emission helpers now fail closed on journal errors instead of paying from an empty in-process cache. 0032 unique digest can reject sloppy test rows that reused one digest; production freeze identities are already unique. No chain submit. Eval image pin is unchanged (candidate digest is still not a prod pin).

Naming

I did not rename BASE_* environment variables, deployed host paths
(/opt/base, /run/base, …), GHCR baseintelligence/base package names, or
base-*-v1 cryptographic domain tags, unless this PR’s purpose is a coordinated
cutover documented in docs/NAMING.md.

  

[echobt]

Review (Développeur)

Verdict: looks directionally right for the payout-journal + listed-op FLOPs goal. Do not merge to main without Mathis go (base is the checkpoint branch; no deploy / set_weights).

What looks good

• Fail-closed boot on list_durable() / snapshot_durable — no silent empty store.
• store_runs / store_scores return Result instead of swallowing into {}.
• Migration 0032 unique submission_digest for idempotent retries (0031 sequence note is clear).
• Listed-op compute trace + controller twin flop_total refusing unlisted ops / untrusted flops_used.
• Docs honesty: not full recipe reproduction; no automatic emission loop; can_score ≠ payment.

Nits / follow-ups

1. BYOK lock (Mathis 2026-09-05): miner brings the pod; master must not auto-rent Lium. “Enable bounded live Lium ≤$30” as next step conflicts with that lock unless reframed as miner-paid / miner-owned compute. Prefer BYOK submit locator over master rent.
2. Confirm runtime CI green before any merge consideration (contract was green earlier; runtime was still pending).
3. Concurrent proof_topic_run last-writer-wins is called out — fine for v1; track if multi-validator scoring lands.
4. Stack: PR targets checkpoint/proof-production-readiness-20260907, not main — rebase/PR-to-main plan still needed when Mathis wants it.

Greptile: leave for bot; I won’t self-merge.

[cursor]

@greptileai review

[greptile-apps]

Greptile Summary

This change adds FLOP trace accounting and submission-digest uniqueness, but three failures need correction before merge: unknown operations can undercount compute, controller-side validation is missing the evidence required to recompute the eval total, and historical duplicate digests can stop an upgrade.

Confidence Score: 2/5

Not safe to merge: compute-budget evidence can be understated, controller verification is not independently enforceable, and valid existing database data can prevent deployment upgrades.

Three independently reproduced failures affect compute-evidence integrity and upgrade reliability.

Files Needing Attention: eval/src/proof_eval/compute_trace.py, eval/src/proof_eval/cli.py, crates/proof-research/src/flops.rs, and crates/db/migrations/0032_proof_submission_digest.sql.

T-Rex Logs

What T-Rex did

• T-Rex produced proofs for several posted P1 findings, each supported by corresponding review comments and accompanied by shell-script and Python artifacts.
• A general-contract-validation-proof for unknown-operation-zero-flops shows an executable reproduction script and before/after runtime logs.
• A general-contract-validation-proof for eval-trace handoff documents the Python-based evaluation flow and the verification of harness and metrics in the logs.
• A general-contract-validation-proof for proof-submission-digest shows the reproduction script and log details including a PostgreSQL failure and reconciliation step.

_{Ran code and verified through T-Rex}

Comments Outside Diff (3)

1. General comment

   Unknown compute operations are silently charged zero FLOPs

   • Bug
     • Confirmed at eval/src/proof_eval/compute_trace.py:128-130: _looks_compute(name) only rejects names matching a limited heuristic, then op_flops returns 0 for every other unknown name. The equivalent Rust path is crates/proof-research/src/flops.rs:44-47 and :149-150. Runtime evidence shows both verifiers accept a valid aten::mm trace totaling 48 FLOPs and return the identical 48 after adding a large unknown aten::linear operation.
   • Cause
     • The allowlist is enforced only conditionally on _looks_compute; unrecognized operation names fall through to the generic zero-cost branch rather than being rejected. saw_compute is satisfied by the listed matrix multiplication, allowing the whole mixed trace to pass.
   • Fix
     • Use a fail-closed trace contract: reject every operation not explicitly classified as a known zero-cost/non-compute operation or assigned a published FLOP cost. Apply the identical rule to Python and Rust, and add parity tests for aten::linear and arbitrary unknown operation names mixed with listed work.

   _{Ran code and verified through T-Rex}

2. General comment

   Evaluation trace and recomputed FLOP count are discarded before controller handoff

   • Bug
     • The eval CLI verifies the evaluation compute trace locally but pops both compute_trace and eval_flops from the harness before serializing the ProofEvalDocument. Rust subsequently deserializes only HarnessMetrics, whose accepted document validation does not receive or independently recompute this evidence. Thus the emitted clean verdict's rationale claiming independent recomputation is not enforceable by the controller.
   • Cause
     • _score destructively calls harness.pop("compute_trace") and harness.pop("eval_flops") at eval/src/proof_eval/cli.py:124-125; the Rust ProofEvalDocument/HarnessMetrics schema contains no retained trace or evaluation FLOP field, and proof_research::flop_total has no production caller.
   • Fix
     • Retain the verified trace and its claimed count in the emitted document (preferably a dedicated evidence field), extend the Rust metrics/document schema to require them, and invoke proof_research::flop_total during controller-side document validation, rejecting absent, malformed, or mismatched evidence.

   _{Ran code and verified through T-Rex}

3. General comment

   Migration cannot upgrade databases with duplicate historical submission digests

   • Bug
     • At crates/db/migrations/0032_proof_submission_digest.sql:7-9, the migration creates a partial unique expression index directly. Against the representative pre-migration 0031 schema containing two valid rows whose JSON documents both have submission_digest: "duplicate-digest", PostgreSQL rejected the exact statement: could not create unique index "proof_submission_freeze"; the index count remained zero.
   • Cause
     • The migration adds uniqueness without first detecting, deduplicating, or otherwise reconciling existing non-empty document->>'submission_digest' values.
   • Fix
     • Before creating the unique index, add an explicit, deterministic duplicate-reconciliation migration (or block with a clear preflight query and documented operator repair) that preserves the chosen canonical submission and removes or rewrites conflicting digest values. Then create the index.

   _{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "feat(proof): snapshot journal payouts an..." | Re-trigger Greptile}

[greptile-apps]

Reject Unknown Operations

The verifier accepts unrecognized operation names as zero-cost once a known compute operation is present. A trace containing a 48-FLOP aten::mm plus a large aten::linear is accepted with the same 48-FLOP total in both the Python verifier and Rust implementation. This allows submitted training evidence to understate compute consumption and pass the compute-budget gate with an invalid total.

Artifacts

Evidence from the check

• This script runs the Python verifier and Rust controller parity implementation against baseline and mixed traces; it demonstrates the executable reproduction used for the finding.

Command output from the check

• The baseline run invoked both implementations with only `aten::mm`; each returned 48 FLOPs with exit code 0, establishing the comparison total.

Command output from the check

• The reproduction run added a large `aten::linear` operation to the 48-FLOP matrix multiplication trace; both implementations still returned 48 and exited successfully, confirming understated totals.

_{Ran code and verified through T-Rex}

[greptile-apps]

Keep Controller Trace Evidence

The eval CLI verifies compute_trace locally, then removes both it and eval_flops before serializing the metrics document. The controller therefore receives neither the operation list nor the claimed total, and has no production call to recompute FLOPs. A locally produced metric can be accepted without the intended independent controller verification.

Artifacts

Evidence from the check

• This executable invokes the repository CLI scoring handoff with a traced harness payload and inventories Rust `flop_total` references, showing the exact evidence expected before serialization.

Command output from the check

• The executed CLI output shows both trace and FLOP count were present before encoding but absent from the emitted harness, proving the controller receives neither.

Command output from the check

• The executed Rust test command passed all three `proof-research` FLOP tests, showing the recomputation function exists while the handoff probe shows no production caller.

_{Ran code and verified through T-Rex}

[greptile-apps]

Reconcile Existing Digests

If an upgrading database contains historical rows with the same non-empty submission digest, this migration's unique-index creation fails. The previous schema permits those rows; applying the migration to two valid duplicate records leaves the index absent and aborts the upgrade. Reconcile or explicitly reject duplicate groups before creating the unique index.

Artifacts

Evidence from the check

• Runs PostgreSQL 16, applies the actual 0031 and 0032 migrations, and compares duplicate-data failure with reconciled-data success; it confirms that unreconciled duplicates block the upgrade.

Command output from the check

• Captured output from executing the reproduction script shows the duplicate-key index failure and then successful index creation after one duplicate is removed; it confirms the reported upgrade blocker.

_{Ran code and verified through T-Rex}