Expand issue taxonomy with underconstraint and implicit_assumption - #7
Open
uriariel wants to merge 1 commit into
Open
Expand issue taxonomy with underconstraint and implicit_assumption#7uriariel wants to merge 1 commit into
uriariel wants to merge 1 commit into
Conversation
Add two new IssueKind values that the Distinguisher can raise: - underconstraint: spec is too weak and admits trivial/broken implementations (e.g. a sort spec satisfied by return input) - implicit_assumption: spec silently depends on an unstated property of inputs (e.g. assuming no duplicates without saying so) These attack patterns come from the Kimi research's structured taxonomy and address the most impactful gaps in the current 4-kind system. The distinguisher prompt is updated with concrete examples of each pattern. Co-Authored-By: Oz <oz-agent@warp.dev>
There was a problem hiding this comment.
Pull request overview
Expands the distinguisher’s issue taxonomy to better capture spec failure modes (weak specs that admit degenerate solutions, and specs that rely on unstated assumptions), aligning the schema, prompting, and tests.
Changes:
- Added
underconstraintandimplicit_assumptionto theIssueKindschema. - Updated the distinguisher prompt to describe and exemplify the two new issue kinds.
- Added schema JSON round-trip tests for both new kinds.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/idcs/schemas.py | Extends IssueKind Literal[...] to include two new issue categories. |
| prompts/distinguisher_v0.md | Documents the new issue kinds with definitions/examples for the LLM prompt. |
| tests/test_schemas.py | Adds round-trip tests ensuring the new kinds serialize/deserialize correctly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+56
to
+63
| IssueKind = Literal[ | ||
| "gap", | ||
| "ambiguity", | ||
| "contradiction", | ||
| "over_constraint", | ||
| "underconstraint", | ||
| "implicit_assumption", | ||
| ] |
Owner
|
Small follow-up before merging: kind: Literal["gap", "ambiguity", "contradiction", "over_constraint"]Worth extending it in this PR to include Generated by Claude Code |
aimir
pushed a commit
that referenced
this pull request
May 24, 2026
The default mutator system prompt didn't know whether it was mutating a generator or distinguisher — it just got asked for "improvements". Mutations were necessarily generic and often resembled paraphrases. Two changes: 1. mutate.py — new default system prompt explains both roles, the full issue taxonomy (including the newly-added underconstraint and implicit_assumption kinds from PR #7), and explicit rules: preserve the role's function, address feedback, be substantive (not a paraphrase), keep markdown structure. mutate() gains a `role` kwarg that's included in the user message. 2. coevolve.py — _summarize_feedback() now returns role-specific diagnostics. The generator sees avg benchmark, avg type-1 raised against it, avg spec complexity penalty. The distinguisher sees avg type-1 raised, avg actually fixed (the accepted-reject rate), avg type-2 raised, avg dismissed, avg useful clarification rate — all the levers it can pull. Each feedback ends with a directive pointing at the role's specific reward terms. Both _init_population (seed mutations) and _evolve_population (per-elite mutation) now pass role=role to mutator.mutate(). No new LLM calls; same call count.
21 tasks
Owner
|
Included in #10, that also fixes a lot of the reviewed issues here |
aimir
pushed a commit
that referenced
this pull request
May 25, 2026
These were the four unresolved-but-still-relevant comments after the cherry-pick of #6/#7/#9 into this branch: - rewards: type2_dismissed_count no longer fires for route=user issues with suggested_question=None. Those issues can't be routed to the user-proxy in run_episode, so penalizing D for the missing answer was unfair. Added regression test. (PR #6 review) - telemetry: create_run_dir now uses mkdir(exist_ok=False) inside a try/except FileExistsError loop. Two runs starting in the same second from concurrent processes no longer race on exists()->mkdir(). (PR #6 review) - design.md: IssueKind list synced to the six kinds actually in schemas.py (added underconstraint and implicit_assumption). (PR #7 review) - 09-sort-underconstraint gold spec: clarified that the returned list is a new object, not the input — both via a new postcondition and by rephrasing the edge cases that previously read as "returns itself". (PR #9 review)
aimir
added a commit
that referenced
this pull request
May 25, 2026
…10) * Expand issue taxonomy with underconstraint and implicit_assumption Add two new IssueKind values that the Distinguisher can raise: - underconstraint: spec is too weak and admits trivial/broken implementations (e.g. a sort spec satisfied by return input) - implicit_assumption: spec silently depends on an unstated property of inputs (e.g. assuming no duplicates without saying so) These attack patterns come from the Kimi research's structured taxonomy and address the most impactful gaps in the current 4-kind system. The distinguisher prompt is updated with concrete examples of each pattern. Co-Authored-By: Oz <oz-agent@warp.dev> * Add vulnerability benchmark tasks with known spec weaknesses Four new seed tasks (09-12) specifically designed to test whether the Distinguisher catches common spec failure patterns: 09-sort-underconstraint: sort spec missing 'permutation of input' 10-divide-implicit-assumption: division without addressing b==0 11-unique-preserve-order: dedup without specifying order preservation 12-clamp-contradiction: range clamping without handling low>high Each task includes a 'known_weakness' field documenting the expected spec failure type and the naive spec gap the Distinguisher should surface. These complement the existing 8 seed tasks (simple algorithmic tasks) with adversarial evaluation targets — tasks where a naive Generator will produce specs that look reasonable but have critical gaps. Co-Authored-By: Oz <oz-agent@warp.dev> * Add coevolution optimizer scaffolding * Fix optimizer lint issues * rewards: key type-1 fixes by (kind, location) only Previous key was (kind, route, location, description). description is free-text LLM output that legitimately varies when D re-flags the same underlying gap — including it lets G score a "fix" just because D rephrased. Route is excluded too: D moving the same gap from generator-routed to user-routed isn't a fix; the gap is still there. Three new tests pin the contract: rewording doesn't count, actual fixes still count, route changes don't count. * rewards: penalize type-2 inflation past a per-episode cap Without a cap, design.md's worry holds: small benchmark wins per question tempt D to ask more. The naïve dismissed-question penalty is proportional and easy to outrun. RewardWeights gains max_type2_per_episode (default 5) and excess_type2_penalty (default 1.0, same magnitude as alpha so excess questions can't be made profitable by being slightly useful). Penalty applies only to D — G can't control how many questions D raises. Two tests pin the contract: G's reward is unaffected at any question count; D's reward drops by exactly (count - cap) * excess_penalty above the cap, and not at all at or below it. * coevolve: thread no-spec baseline_score into the reward signal Without this, coevolve was passing baseline_score=None to compute_reward_breakdown, which forces useful_clarification_rate to 0 on every candidate evaluation. The delta term that's supposed to give D credit for surfacing useful questions never fires. Fix: compute the per-task no-spec baseline once at the start of coevolve() (one LLM call + one score() per task, vs the N_tasks × N_candidates × N_epochs × 2_roles recompute it'd take to do this inside the inner loop). Pass the dict through _evolve_population → _evaluate_candidate → compute_reward_breakdown as baseline_score=baselines.get(task.id). This is still the *naïve* attribution (Δ between with-spec and no-spec, divided equally across all user answers). Per-question counterfactual attribution from design.md remains a follow-up — it would multiply LLM cost by O(type-2 questions per episode). Three new tests on compute_reward_breakdown: - baseline_score=None ⇒ rate is 0 (current behavior preserved) - baseline_score=0.5, benchmark=0.8, 1 clar ⇒ rate is 0.3 - baseline set but no clarifications ⇒ rate is 0 (no div-by-zero) Optional baseline_scores kwarg on coevolve() lets callers (or tests) inject pre-computed baselines. * coevolve: sample opponent once per candidate, not per task Per-task opponent sampling injects noise that isn't meaningful: we want this candidate's average reward against *one* sampled opponent, not against a random walk through the opponent population. design.md calls for sampling "D from P_D" (singular) per candidate evaluation. One-line move out of the per-task loop. No behavioral change to the external API. * population: diversity guard on top_k elite selection Pure elitism collapses populations onto k clones over a few epochs — the highest-rewarded prompt and its near-paraphrases occupy every elite slot, and the population effectively becomes a single point. Mutation on top of that point just orbits it. top_k now walks candidates in descending-reward order and skips any whose prompt is too similar (SequenceMatcher.ratio() ≥ threshold, default 0.92) to one already selected. If diversity rejects more than the pool can spare, fall back to highest-reward to still return k — the caller's contract is "give me k candidates", not "give me only the diverse ones". Three new tests: - near-duplicates filtered out - fallback fills k when diversity starves the pool - threshold=1.0 only rejects exact duplicates (escape hatch) * mutator: role-aware system prompt and richer feedback The default mutator system prompt didn't know whether it was mutating a generator or distinguisher — it just got asked for "improvements". Mutations were necessarily generic and often resembled paraphrases. Two changes: 1. mutate.py — new default system prompt explains both roles, the full issue taxonomy (including the newly-added underconstraint and implicit_assumption kinds from PR #7), and explicit rules: preserve the role's function, address feedback, be substantive (not a paraphrase), keep markdown structure. mutate() gains a `role` kwarg that's included in the user message. 2. coevolve.py — _summarize_feedback() now returns role-specific diagnostics. The generator sees avg benchmark, avg type-1 raised against it, avg spec complexity penalty. The distinguisher sees avg type-1 raised, avg actually fixed (the accepted-reject rate), avg type-2 raised, avg dismissed, avg useful clarification rate — all the levers it can pull. Each feedback ends with a directive pointing at the role's specific reward terms. Both _init_population (seed mutations) and _evolve_population (per-elite mutation) now pass role=role to mutator.mutate(). No new LLM calls; same call count. * coevolve: optional held-out val split for overfit monitoring Same training tasks evaluating every epoch's candidates means the optimizer can latch onto task-specific quirks. Adds an optional val_tasks kwarg to coevolve(): when provided, the best-of-population G and D are evaluated against val_tasks at the end of each epoch and the result is written to metrics.jsonl as a `split: "val"` row. Val is monitor-only — it never feeds selection. If a run's training reward climbs while val stays flat or drops, that's the overfit signal. scripts/train.py gains --val-fraction (default 0 = off; typical 0.2). Splitting uses the same seed as the rest of the run for reproducibility. _compute_baselines now covers both train and val tasks in one pass so useful_clarification_rate works on the val side too. * telemetry: config.json snapshot + prompt hashes on Trace Two interpretability fixes for run output. D.11 — config.json snapshot. At the start of each coevolve() run we now write run_dir/config.json with: model id, full RewardWeights, full CoevolveConfig, train task ids, val task ids, and the per-task no-spec baselines. Without this snapshot the metrics.jsonl rows are uninterpretable a few days later (what model? what weights? what split?). D.12 — prompt hashes on Trace. Adds optional generator_prompt_hash / distinguisher_prompt_hash fields to the Trace schema. The optimizer stamps both onto each trace before writing. metrics.jsonl already had prompt_hash for the per-task rows; this lets you also grep traces.jsonl by hash and correlate "this trace ↔ that candidate" without joining two files. Both fields default to None so cold-start (non-coevolution) traces stay valid. Shared _hash_prompt() helper replaces the inline hashlib call in _trace_metrics. * llm: --max-llm-calls budget tracker with hard ceiling Coevolution runs are expensive — pop_size × epochs × tasks × roles multiplies fast, and an open-ended config can burn through a model budget overnight. Adds a hard ceiling. - LLM gains max_calls (default None = unlimited) and calls_made. - _with_retry is now a method (LLM._with_retry) that increments calls_made on every attempt — including failed 429/5xx retries, because those are also billable. - check_budget() runs before each attempt; raises BudgetExceededError once the count would exceed max_calls. Caller's choice whether to catch it. - scripts/train.py: --max-llm-calls flag wires the ceiling, catches BudgetExceededError, exits 2 with a clear message. Final summary always prints LLM calls used so you can size budgets for the next run. Three unit tests with a mocked OpenAI client confirm the counter increments, the budget blocks, and unset budget is unlimited. Doesn't double-count complete_typed's text-fallback path: complete() delegates to _with_retry, which is the single point of truth for the counter — the parent complete_typed has already counted its parse() attempt by the time it falls back. * coevolve: optional cheaper mutator_llm Mutating prompts doesn't need the strongest reasoning model — the mutator is just rewriting + restructuring text. Adds an optional mutator_llm kwarg to coevolve(); when set, Mutator() uses it instead of the main llm. scripts/train.py reads IDCS_MUTATOR_MODEL env var (e.g. "anthropic/claude-haiku-4-5") and constructs a separate LLM for the mutator if set. With pop_size=8 and epochs=5 the mutator burns through ~50-100 calls per run; swapping a cheaper model there significantly trims cost without affecting the evaluation signal (the main model still drives every G / D / coder / user_proxy call). * scripts/inspect.py — read coevolution run output Browses a run directory's config.json + metrics.jsonl + traces.jsonl into something a human can actually look at. Three modes: - default (no flags): config summary, per-epoch G/D best/avg reward table including val numbers when present, per-task reward summary. - filter mode (--epoch/--task/--role): print matching metrics.jsonl rows as JSON for piping into jq. - trace mode (--trace TASK_ID [--prompt-hash H]): pretty-print full Trace objects for one task, optionally narrowed by prompt hash. design.md called for scripts/inspect.py in Phase 3; this is the v0. Useful to confirm a run did what you expected and to dig into specific candidate × task pairs by hash. Smoke-tested on a synthetic run dir — config print, epoch table, and per-task rollup all render correctly. * coevolve: per-stage progress logging Until now the only run-time signal was raw httpx POST lines from the openai client, which makes a 300-call run feel opaque. coevolve now logs at the natural boundaries: - start: pop / elite / epochs / max_turns / task counts - "computing no-spec baselines..." with task count - "initial populations ready" after seed mutations - "=== epoch N/M ===" between epochs - per-role best/avg reward after each evolution step - "candidate N/M" inside _evolve_population - "val eval (N tasks)..." before val evaluation Each log line that comes after an LLM call appends [<n> calls] using LLM.calls_made when available — gives a running call counter for free without a separate tracker. Falls back gracefully if the LLMClient duck-type doesn't expose calls_made (e.g. a FakeLLM in tests). * coevolve: cap elite_size below population_size Footgun discovered during the first real run: passing --pop-size 3 with the default elite_size=3 means every candidate is elite, so the mutation-fill loop (while len(new_members) < pop_size) never fires. The population is identical from epoch 1 onward — what looked like "G regressed 0.889 → 0.878" was actually noise from task-sample variance across two evaluations of the same 3 prompts. If elite_size >= pop_size at start, log a WARNING and cap to max(1, pop_size - 1) so at least one mutation slot exists. The user gets explicit notice rather than a silently dead evolution. Default elite_size stays at 3 (fine for the typical pop_size of 8+). Could also have raised, but a warning + auto-cap is friendlier than a hard failure on a config that mostly works. * Rename scripts/inspect.py → scripts/inspect_run.py scripts/inspect.py shadows the stdlib ``inspect`` module. When the script is run, Python puts the script's directory first on sys.path, so any subsequent ``import inspect`` (e.g. from pydantic / openai) finds our file instead of the stdlib and dies with "AttributeError: module 'inspect' has no attribute 'signature'". Classic Python naming footgun. Renaming fixes it. * log: quieter httpx, louder progress Three logging tweaks based on running the first real training loop. 1. Suppress httpx INFO. The "POST .../v1/chat/completions 200 OK" lines were the only progress signal pre-Stage-D; with the new coevolve log lines they're just noise. Bump httpx logger to WARNING in train.py. 2. After computing no-spec baselines, log min/mean/max so you can see the floor without having to open inspect_run.py. 3. Inside _evaluate_candidate, log each task's outcome live: "task 3/4 vuln/10-divide-implicit-assumption: turns=2 benchmark=0.83 reward=0.92 [187 calls]". Used to only see "candidate N/M" at the outer boundary; with 4 tasks per candidate that's a 30-second gap of silence per row. Now there's a heartbeat every ~5 seconds. 4. Log val results inline right after _evaluate_on_val finishes, instead of writing them to metrics.jsonl and disappearing. You see benchmark / rG / rD per epoch as they happen. * Complete PR10 telemetry and coevolve test * Address pre-merge review comments from PRs #6, #7, #9 These were the four unresolved-but-still-relevant comments after the cherry-pick of #6/#7/#9 into this branch: - rewards: type2_dismissed_count no longer fires for route=user issues with suggested_question=None. Those issues can't be routed to the user-proxy in run_episode, so penalizing D for the missing answer was unfair. Added regression test. (PR #6 review) - telemetry: create_run_dir now uses mkdir(exist_ok=False) inside a try/except FileExistsError loop. Two runs starting in the same second from concurrent processes no longer race on exists()->mkdir(). (PR #6 review) - design.md: IssueKind list synced to the six kinds actually in schemas.py (added underconstraint and implicit_assumption). (PR #7 review) - 09-sort-underconstraint gold spec: clarified that the returned list is a new object, not the input — both via a new postcondition and by rephrasing the edge cases that previously read as "returns itself". (PR #9 review) --------- Co-authored-by: uri ariel <uri.ariel@granulate.io> Co-authored-by: Oz <oz-agent@warp.dev> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Amit Saroussi <19860280+ssaroussi@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add two new
IssueKindvalues the Distinguisher can raise:underconstraint— the spec is too weak: a trivial or broken implementation could satisfy every stated constraint (e.g. a sort spec satisfied byreturn inputunchanged)implicit_assumption— the spec silently depends on an unstated property of inputs or environment (e.g. assuming no duplicates without saying so)Why
The current 4-kind taxonomy (gap, ambiguity, contradiction, over_constraint) misses the two most impactful spec failure patterns identified in the Kimi research.
underconstraintis particularly critical — it's the root cause of "verification theatre" where specs look strong but admit degenerate implementations.implicit_assumptioncatches hidden dependencies that lead to production failures.Changes
schemas.py: ExpandedIssueKindliteral typeprompts/distinguisher_v0.md: Added concrete descriptions and examples for both new kindstests/test_schemas.py: Added round-trip tests for both new issue kindsWarp conversation
Co-Authored-By: Oz oz-agent@warp.dev