Repository navigation
chore(harness): polish — eval job in CI, docs, plan supersession, 0.31.0 (CLEAN-158) - #31
Open
maksymhryzodub-prog wants to merge 4 commits into
Open
maksymhryzodub-prog wants to merge 4 commits into
maksymhryzodub-prog wants to merge 4 commits into
Conversation
…1.0 (CLEAN-158) Phase 9 of the ECC harness principles (specs/001-ecc-agent-principles, T085-T090), plus the third security-review fix set (US5/US6). - CI: a separate `eval` job on main runs `bun run eval`; model-backed sets report "pending" without an LLM_API_KEY secret, a missed threshold fails the job, and the image build does not depend on it. - Docs: guard/instinct/session/skill/task rows in the agent slice README, loop row in the runtime README; IMPROVEMENT_PLAN marks #4, #6, #10, #15 as superseded by the CLEAN-150 phases; quickstart-results.md records what each scenario is covered by and which steps remain manual. - Version 0.31.0. - Security review: workers are bounded by effect, not by a name list — without tools.delegate.allowWrite they get read tools (plus per-call tools such as file/http whose write calls the loop refuses), so a new or MCP write tool is excluded by default; the verifier prompt fences tool output and the draft as data and a rewrite that balloons or adds a link the draft never had is dropped in favour of the draft marked unverified. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ill files (CLEAN-158) gray-matter evaluates `---js` / `---javascript` frontmatter as code. Rule files are operator-authored, instinct files runtime-written and SKILL.md files model-written via skill_write — all live in the agent directory, so none may carry executable frontmatter. A shared safeMatter() refuses any language tag on the opening fence and disables every non-YAML engine; the three gateways use it, and a spec proves `---js` is rejected without running. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
maksymhryzodub-prog
added this pull request to stack #32
October 12, 2026 15:50
…LEAN-158) Seven sections mirroring the user stories, run against a live agent over a channel. Marks which scenarios also have automated counterparts and which exist only here (protected-rule override, tool_describe, instinct conflicts, hard-limit compaction, quiet tool failures, verifier injection, worker concurrency and history isolation, audit trail, frontmatter refusal). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lts (CLEAN-158) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
evaljob onmain(bun run eval). Offline sets (guardrails) always run; model-backed sets (skill selection, corrections, completion claims) need anLLM_API_KEYsecret and reportpendingwithout it. A missed threshold fails the job; the image build does not depend on it, so a flaky eval never blocks a release. OptionalLLM_EVAL_MODELrepo variable picks the model.IMPROVEMENT_PLAN.mdmarks feat(telegram): group chat support + channel source context in system prompt #4 Curator, feat(chat): realtime session-activity signals (Phase 3) #6 Delegation, feat(mcp): oauth MCP servers — headless provider + connect tool (CLEAN-75) #10 Safety pipeline (in part), fix(llm): send the Claude Code identity line only to OAuth clients (CLEAN-107) #15 Lifecycle hooks as superseded by the CLEAN-150 phases,README.mdcarries Guardrails / Verified completion / Delegation / Learning / compaction sections and the new commands.specs/001-ecc-agent-principles/checklists/quickstart-results.md: every quickstart scenario mapped to the spec or eval that covers it; the steps that need a live channel are listed as manual and were not run.0.31.0(no tag pushed —release.ymlruns onv*tags).Security-review fixes (US5/US6)
isDelegateAllowed: withouttools.delegate.allowWrite(defaultfalse) a worker gets onlyeffect: "read"tools plus per-call tools (file,http); the loop refuses a worker's write call per call (Workers are read-only: …, recorded as blocked). A new or MCP write tool is therefore excluded by default instead of needing to be listed. The name list stays as belt and braces.<<<DATA … >>>, delimiter neutralised inside), the model is told to ignore instructions found there, the draft is bounded; a rewrite longer than ~3× the draft or carrying a URL the draft did not have is dropped and the draft goes out(unverified).State of the whole feature
bun test: 309 pass, 1 pre-existing Windows-only failure (telegramFilepath separator).tsc --noEmit: clean apart from the pre-existingbridleAttachments.spec.ts.Merge order: #24 → #25 → #26 → #27 → #28 → #29 → #30 → this; each later PR retargets to
mainas its base merges (or rebase if squash-merging).Jira: https://dreamvention.atlassian.net/browse/CLEAN-158 · parent https://dreamvention.atlassian.net/browse/CLEAN-150
Manual acceptance
specs/001-ecc-agent-principles/checklists/manual-acceptance.mdscripts the live-agent run: seven sections mirroring the user stories, with the scenarios that have no automated counterpart called out (protected-rule override, tool_describe, instinct conflicts, hard-limit compaction, quiet tool failures, verifier injection, worker concurrency and history isolation, audit trail, frontmatter refusal).🤖 Generated with Claude Code