fix(simulated-user): say which, when asked again after a yes (version 3)

[that-guy-wade]

Summary

The simulated user approves with "yes, go ahead and {action}", naming the one approved action. When an assistant showed replies to two buyers and the user said "yes, go ahead and send a reply to a buyer", assistants that would not guess asked "Maria, Jayden, or both?". The user answered with the same yes, again and again. In the published runs this stalled 10 conversations, up to 57 turns. Several ended with a run waiting for a person.

Now, asked right after its yes which of several of that action to do, the user says "yes, go ahead and {action}, every one you showed me", if doing every one meets its approval rule. Otherwise it gives the plain yes as before. The simulated user's version is 3.

Changes

• SimulatedUser._every: there are three guards. Each is needed, judging by the replay below.
  1. The user's last decision was a yes to the same action, alone or given together with a fact, so the first yes is unchanged.
  2. The reader picks "which of several people or items" over "whether to do it" and "something else". That rules out button yes/no ("Send it / Don't send") and alternatives ("which card?").
  3. The rules judge confirms that doing every one offered meets the rule, so "every one" never approves something the rule forbids.
• _allowed(every=True) asks that last question. The proposal question it already asked keeps its text and context, so its key and saved answers are unchanged.
• FIXED["approve_every"], approval(operation, every), VERSION = 3.
• Docs: gold-behaviour.md U10, the run-record.md example. experiments/probe.py records each replayed reply's text.

Testing

Replay of every simulated-user decision in all 1,489 published runs (2,521 decisions), official reader and judge:

• Without guards 1–3: the "every one" yes fired on 125 of 333 approvals. That included button yes/no questions, and "cancel JobFlow? and Sketch3D too?", where "every one" would have approved what the user's rule forbids.
• With the guards: it fires 24 times.
  • 21 are the loop: Pally, Grok Bot and Muse asking "Maria, Jayden, or both?" / "Marco, Kyle, or both?".
  • 2 are offers to send the same reply to the other buyer, where each reply holds the floor.
  • 1 is an assistant asking where to send its single draft. The yes then still covers only that draft.
• No other decision changes (1 unrelated decision differs between replays).
• Re-running the replay after the final refactor gives identical decisions with no new judge calls.
• Tests:
  • test_asked_which_after_a_yes_the_user_says_every_one fails on main.
  • test_the_yes_covers_every_one_only_if_all_of_them_meet_the_rule fails if guard 3 is skipped.
  • test_a_first_choice_gets_the_plain_yes checks the first yes is unchanged.
  • test_a_yes_given_with_a_fact_counts_as_the_yes_before_the_choice covers a yes given together with a fact.
• Suite: uv run pytest -q gives 668 passed. ruff format --check is clean.

Size

+168 / −25. CC: _every 7; _allowed 7 → 9 (the every question); _answer unchanged at 5.

Risks

• Comparability: version 3 changes conversations only where an assistant asks again which of several to do after a yes. The version is recorded in every run.
• Not yet measured live on the products that loop. This replay shows the user's new answer. Whether each assistant then sends both is its own behaviour, to be checked in a live run.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; the combined-approval fix keeps the existing approval checks.

Summary

Version 3 lets the simulated user answer a repeated “which ones?” question with “every one you showed me” when all offered actions meet its rule.

• A repeat choice gets an all-items yes only when every item fits the user's rule.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Proposal meets the user rule] --> B{Previous reply approved the same action?}
  B -->|No| C[Plain yes]
  B -->|Yes, alone or with a fact| D{Assistant asks which ones?}
  D -->|No| C
  D -->|Yes| E{Every offered item meets the rule?}
  E -->|No| C
  E -->|Yes| F[Yes, every one shown]

Loading

_{Reviews (2) · Last reviewed commit: "fix(simulated-user): a yes given with a ..." · Reviewed by Greptile}