fix(proof-harvest): log stdout_tail on PROOF_EVAL_OK refuse - #233
Conversation
Live harvest discarded pod stdout when the eval image omitted PROOF_EVAL_OK, so operators only saw a blind refuse. Keep the last 8 KiB via harvest-pod's truncate_tail on the warn. Co-authored-by: Mathis <echobt@users.noreply.github.com>
|
@greptileai review |
|
@greptileai review — please complete; CI already green (fmt/clippy/test + CodeQL). Needed for merge of stdout_tail harvest fix. |
Greptile SummaryThis change retains a bounded, UTF-8-safe evaluator stdout tail when a proof evaluation fails to emit its success marker. One non-blocking coverage gap remains: the new refusal test does not verify that the diagnostic warning contains the retained stdout tail. Merge safety: safe to merge, though direct warning capture coverage would prevent this observability behavior from silently regressing. Confidence Score: 4/5Safe to merge: the confirmed issue affects regression coverage for diagnostics rather than proof-harvest behavior, returned errors, or persistence. One confirmed non-blocking coverage concern remains. The focused check showed that the refusal test passes both before and after removal of the stdout-tail warning field. Files Needing Attention: crates/proof-harvest/src/lib.rs: add a test that captures and asserts the warning's stdout-tail field.
 T-Rex LogsWhat T-Rex did
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| async fn harvest_refuses_stdout_without_ok_marker() { | ||
| let recs = synthetic_holdout(STRATUM_SIZE, 1); | ||
| let topic = harvest_topic(&recs); | ||
| let pod = StdoutPod::new("refused: no model: Qwen/Qwen3.8-0.6B\nexit=2\n"); | ||
| let harvest = LiumProofHarvest::new( | ||
| pod.clone(), | ||
| HarvestLimits::default(), | ||
| vec!["ssh-ed25519 AAAAtest proof".into()], | ||
| ) | ||
| .with_judge_api_key(Some("sk-live-not-a-real-secret".into())); | ||
| let err = harvest | ||
| .score( | ||
| &harvest_pin(), | ||
| &topic, | ||
| &harvest_offer(), | ||
| "digest-abcdef", | ||
| "artifact", | ||
| &recs, | ||
| "claim", | ||
| ) | ||
| .await | ||
| .expect_err("no ok"); | ||
| assert!( | ||
| matches!(err, EvalError::Backend(ref m) if m.contains(OK_MARKER)), | ||
| "{err}" | ||
| ); | ||
| assert!(*pod.booted.lock().expect("booted")); | ||
| } |
There was a problem hiding this comment.
This non-blocking test coverage gap verifies only the returned error and that the pod booted; it never observes the warning or its stdout_tail field. Removing the stdout-tail diagnostic leaves this test passing, so the new failure diagnostics can regress without detection. Add log capture and assert the refusal text is included in stdout_tail; otherwise, future evaluator failures may again lack the information needed to diagnose them.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Artifacts
- The executed script runs the supplied Rust test before and after removing the warning diagnostic in a disposable worktree, demonstrating whether the test detects the regression.
- The exact supplied proof-harvest refusal test ran in the unmodified checkout and passed, establishing the baseline.
- The exact supplied refusal test ran after the stdout-tail warning field was removed in an isolated worktree and still passed, proving the test gap.
2 participants
Summary
Live Proof harvest discarded pod stdout when the eval image omitted
PROOF_EVAL_OK, so operators only saw a blind refuse (eval image did not print PROOF_EVAL_OK) and had to re-rent to diagnose. Prod E2E 2026-09-08 hit this: a B200 came up, then harvest refused ~5s after RUNNING because score exited 2 on an invalid HF id (Qwen/Qwen3.8-0.6B).This PR logs a truncated stdout tail (8 KiB) on that refuse via
harvest_pod::truncate_tail(re-export ofprism_lium::truncate_tail). The returnedEvalErroris unchanged.What
PROOF_EVAL_OK,tracing::warn!now includesstdout_tail(last 8 KiB, UTF-8 safe).Out of scope
DEFAULT_PROXY,PROOF_PROXY_MODEL_DIR).PROOF_HOLDOUT_STORE.Greptile
Every PR is reviewed by Greptile before merge. Config:
.greptile/.@greptileai reviewTest plan
cargo test -p proof-harvestcargo fmton the touched cratecargo clippy -p proof-harvest --all-targets -- -D warningsRisk
Operator logs only. No scoring, emission, pin, or miner CVM change. Stdout may contain eval diagnostics (the point of this PR); the 8 KiB cap matches the pod
tail -c 8192 run.logalready appended to harvest stdout.Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags, unless this PR’s purpose is a coordinatedcutover documented in
docs/NAMING.md.