Skip to content

feat(agent_runtime): advisory repeated-evidence ledger alongside repeat_guard - #247

Merged
Zongwei9888 merged 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:rework/evidence-ledger
Sep 27, 2026
Merged

Zongwei9888 merged 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:rework/evidence-ledger

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Description

Adds an advisory count of repeated evidence next to repeat_guard, in the shape suggested when #178 was closed: an advisory layer feeding the guard's existing injection channel, with a real entry point and no hard blocks.

repeat_guard only counts consecutive identical calls. A stall that alternates calls (A, B, A, B, …) never trips it even though the model is learning nothing. This ledger keys on (canonical call signature, result fingerprint) and counts how often the same evidence comes back, regardless of interleaving.

Related Issues

Refs #178 (closed) — reworked per review.

Changes Made

  • core/agent_runtime/evidence_ledger.py (+102): EvidenceLedger.observe(tool_name, arguments, result) -> str | None. Default threshold DEFAULT_NO_PROGRESS_THRESHOLD = 3, matching repeat_guard's default-on posture; bounded at 256 tracked calls with LRU eviction; fires exactly once per distinct result; invalid thresholds fail loudly (bool rejected, ints ≥ 2). Non-text results are canonicalized — the Goal tools return dicts — so a reminder can always be computed and a run never fails because a reminder could not be.
  • core/agent_runtime/runner.py (+54/−7): the ledger rides the existing reminder channel rather than adding one. AgentRunSpec.evidence_ledger_threshold mirrors repeat_call_thresholds; the reminder is a single injected user message plus the same repeat_guard note source used today. Calls already flagged by the tracker are left to the tracker, so the two never double-message. None disables it.
  • tests/test_evidence_ledger.py (+369, 12 tests): threshold semantics and once-only firing; the interleaved stall the guard cannot see; argument-order insensitivity; new evidence restarting the count; bounded tracking with eviction; invalid thresholds; structured results; and four runner-level integration tests (injection exactly where the guard is blind; the guard still owns consecutive repeats; a structured result counts as evidence; disabled means silent).

Checklist

  • Changes tested locally (12 new tests + the runtime suite; ruff check + ruff format --check clean)
  • Unit tests added (if applicable)

Additional Notes

  • Advisory only: nothing is delayed, rewritten or blocked; the decision stays with the model. The ledger sits in front of any hard stop (max_iterations, should_stop_callback) rather than replacing one.
  • The reminder never quotes tool output (asserted in tests), so it cannot echo untrusted content back into the loop.
  • Pre-existing platform failures in tests/test_hooks.py and tests/test_session_end_lifecycle.py were triaged as unrelated (identical results with runner.py stashed) before this was submitted.

repeat_guard only sees *consecutive* identical calls. The other classic
stall interleaves different calls (A, B, A, B, ...) and gets the same
evidence back every time: no call is ever repeated consecutively, so
nothing fires while the run learns nothing.

EvidenceLedger keys on the call *and* its result. For each canonical call
signature it counts the result fingerprints already seen; when one pair
comes back `evidence_ledger_threshold` times (default 3) the call provably
is not making progress and one reminder is emitted.

Same discipline as repeat_guard, which is the point:

- advisory only: nothing is delayed, rewritten, or blocked, and the
  decision stays with the model; the ledger sits in front of any hard stop
  (`max_iterations`, `should_stop_callback`);
- it rides the existing reminder channel (`AgentRunSpec.evidence_ledger_threshold`,
  mirroring `repeat_call_thresholds`, and the same single user message and
  `repeat_guard` note source), so a call the tracker already flagged is
  left to the tracker — one iteration injects at most one reminder per
  call;
- the reminder never quotes tool output, and its state is bounded (256
  calls, LRU);
- results are a dynamic boundary, so a non-text result (the Goal tools
  return dicts) is canonicalized instead of assuming `str`.

`None` disables the layer.

(cherry picked from commit f1b9b2a2ba16299db64c0ad42ae5fa7b0208b060)
@Zongwei9888
Zongwei9888 merged commit 1bb0c93 into HKUDS:main Sep 27, 2026
12 checks passed
@Zongwei9888

Copy link
Copy Markdown
Collaborator

Merged into main as 1bb0c93. Thank you @raymondginger2018-sudo — this is the advisory layer suggested when #178 was closed, riding the existing reminder channel with a real entry point. I checked the two things I was most worried about: tool results are paired with calls by the same ordered zip the runner uses to build the tool messages, so the ledger cannot attribute one call's result to another; and calls repeat_guard already flagged are skipped, so a batch never gets two reminders for one call. The full suite is green with the ledger on by default.

For the record, since it now affects every run: an interleaved poll such as wait_agent returning the same "still running" text three times will earn one reminder, the same way three consecutive identical waits already do under repeat_guard. If that turns out to be noisy in practice, a per-tool exemption would be the natural follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants