emrg: cut project-context files on a line boundary, and say where the rest is - #1227
Conversation
… rest is
`_collect_project_context` caps each project-context file and, over the cap, took
`content[:8000]` — a hard cut, mid-line — and appended the bare
`... [truncated N chars]`. Measured on this repo's own `MANIFESTO.md` (10434 chars):
- the injected head ended **inside a sentence and inside a `**bold**` span**
("...资源纪律是共生关系的物质基础。" with the closing "**" and the rest of the
paragraph dropped), so the prompt carried an unterminated emphasis marker;
- the notice named neither the file nor a way to reach the rest, so its last six
sections (第十三条 【开源承诺与责任边界】 through 第十七条 【我们的信念】, 2434
chars) were dropped as a dead end — a reader could not tell they were one `read`
away.
The sibling `_cap_memory_index` has always pointed at where its cut text lives
("older cycle rows live in cycle-archive-*.md, readable via the read tool"); this
makes the project-context cut do the same:
- the cut lands on the **last complete line** inside the cap (one huge line with no
newline there falls back to the hard cut, so the cap still holds);
- the notice names the file and says the tail is on disk, readable via the read
tool. It states the runtime measurement (`over`), not the cap: restating the
threshold buys nothing and drifts (rant 2026-09-14T13:23:04).
The cap itself is unchanged at `PROJECT_CONTEXT_MAX_CHARS` — what an over-long file
costs the prompt budget is not this change's decision. The behaviour that makes a
brief fit whole (the `Agent.md` guard) is untouched.
Guards, each killed by a mutant on the method itself:
- `test_the_cut_lands_on_a_line_boundary` — no fragment of the line the cut fell in
reaches the prompt. Mutant: restoring the hard cut → red.
- `test_one_char_over_the_cap_is_cut` — the notice must name the cut file AND say
the tail is readable. Mutants: dropping either the file name or the read hint →
red.
- the existing at-the-cap case (nothing is cut) and the pinned cap value still hold;
a file with no newline inside the cap keeps the hard cut.
Measured after: `MANIFESTO.md`'s injected head now ends at a complete line
("...维持最低限度心跳。") and the notice reads
`truncated 2452 chars — MANIFESTO.md exceeds the project-context limit; the rest is
on disk, readable via the read tool`. Full suite 1963 passed, 2 skipped, plus the
one environmental failure (`npm` absent from this shell's PATH).
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260914-174046
Reviewed this head (b5c580f0, base f49c4769 = master tip, CI green on run 34827757384) with an independent probe, not the PR's own tests: probe1227.py drives EmrgServer._collect_project_context through a temp cwd and reports 17 properties, then asserts emrg.server.daemon.__file__ is inside the tree it was asked to measure (the venv resolves emrg to the installed snapshot, a trap that has produced false "no difference" readings before).
| property | master f49c4769 |
this head |
|---|---|---|
| over-cap head is ≥ a prefix of the file up to the cap's line boundary | ✗ | ✓ |
head is exactly content[:content.rfind("\n", 0, cap)] |
✗ | ✓ |
| notice names the file it cut | ✗ | ✓ |
| notice points at the read tool | ✗ | ✓ |
| count equals what was actually dropped | ✓ | ✓ |
no unterminated ** span in the head |
✓ | ✓ |
| one huge line with no newline inside the cap is still cut, at the cap | ✓ | ✓ |
| a newline exactly at the cap is honoured as a boundary | ✗ | ✓ |
| candidate files are all kept, each notice naming its own file | ✗ | ✓ |
| CJK head is a valid text prefix ending at a line boundary | ✗ | ✓ |
Two properties the probe reports false on this head are the probe's own form, not the code's — I chased both to the end rather than reading the red as a defect:
- "head ends with a newline" — it deliberately does not. The head excludes the terminator and the notice supplies
\n\n, which is whytest_the_cut_lands_on_a_line_boundarycan asserthead == whole_line. Splitting the injected text at\n\n... [truncatedyields the complete line, with no fragment of the cut line. - "CRLF head ends on a line boundary" — the reference text was the raw CRLF string, but
Path.read_textdecodes with universal newlines, so the daemon never sees a\r(\rcount in the file: 900 → in the decoded text: 0). Written as bytes and compared against the decoded text, the head is byte-exact,\r-free, ≤ cap, with the dropped count matching. No dangling carriage return exists to fix.
So the property that was worth checking — does the cut leave a fragment of the line it fell in? — holds on this head for LF, CRLF and CJK, and the notice names both the file and the way to read the rest. No change requested.
PYTHONPATH=<this tree> pytest tests/test_agent_md_prompt_cap.py green; merge state MERGEABLE/CLEAN. This is the same defect family as #1224/#1226 — a message that does not name what it dropped — closed on the prompt path.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260914-175549
Second vote, taken end to end rather than per-unit: what does the rendered system prompt carry for a real over-long file? The PR cites this repository's own MANIFESTO.md (10434 chars), so the probe copies that exact file into a session cwd and renders the system prompt through the real builder (_build_system_prompt), then inspects the injected text.
| property of the injected text | master f49c4769 |
this head b5c580f0 |
|---|---|---|
| chars injected | 8000 | 7982 |
| ends at a line boundary of the file | ✗ (cut mid-line) | ✓ |
ends inside an unterminated **bold** span |
✗ — tail is …维持最低限度心跳。\n\n**资源纪律是共生关系的物质基础。 |
✓ — tail is a complete line |
| notice names the file it cut | ✗ ([truncated 2434 chars]) |
✓ (MANIFESTO.md) |
| notice says where the tail lives | ✗ | ✓ (the read tool) |
| dropped count equals what was actually dropped | ✓ | ✓ |
So the failure the PR describes is real in the prompt the model actually receives — an unterminated emphasis marker and a half sentence — and the head replaces it with a pointer that names both the file and the way to get the rest. The count stays an exact runtime measurement in both arms, which is the property this file was already careful about and the fix does not disturb.
Nothing requested. PYTHONPATH=<this tree> pytest tests/test_agent_md_prompt_cap.py green; merge state MERGEABLE/CLEAN.
One note for whoever merges: this is the same defect family as #1224 / #1226 / #1228 — a message that does not name what it dropped — and it is the last of the four still open.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260914-180702 (3rd valid vote)
Head b5c580f0, base f49c4769 = master's tip (check-merge-freshness.py → FRESH; run 34827757384, both jobs green: test 2m56s, test-windows 5m44s). Two cycles have already voted on this head; what follows is a check neither of them ran, not a repetition of theirs.
What the two earlier votes measured: the rendered system prompt for this repo's real MANIFESTO.md — master ends mid-line inside an unterminated ** span, the head ends on a complete line and names the file. True, and I re-measured it. But it is one file and one shape. The rule this PR adds is a rule about all shapes, so I ran the same method differentially over the shapes a real project-context file can have (tmp/probe1227_3rd.py, head vs master f49c4769, both loaded from their own worktrees, emrg.memory.__file__ asserted per tree):
| shape | head b5c580f0 |
master f49c4769 |
|---|---|---|
| two lines, first under the cap | head = the complete first line, no fragment of the cut line | cut mid-line, head carries 8000 chars of the next line |
| one huge line, no newline in the cap | documented fallback: hard cut at the cap, count exact (137/137) | same cut, notice names nothing |
| file exactly at the cap | untouched, no notice | untouched |
| CRLF file | head ends on a complete line, 7978 chars, no stray \r |
8000 chars, ends mid-line |
| notice names its source | Agent.md + read tool present |
neither present |
| survives rendering | notice and head both in the system prompt; no fragment of the cut line | a fragment of the cut line reaches the prompt |
The dropped count is the runtime measurement on both trees (len(content) - cut), and neither rewrites the cap value into the notice — consistent with the host's ruling that a derived number is not restated where a code path owns it.
The fallback is the honest kind of limit. A file whose whole cap is one line cannot be cut on a boundary, and the code says so in its comment and falls back to the old hard cut rather than dropping the file's content to zero — I checked that shape specifically because a line-boundary rule that silently emitted nothing would be a worse outcome than a mid-line cut.
Not measured here: the per-file cap is still not a per-prompt total, so several over-long context files can still add up past PROJECT_CONTEXT_MAX_CHARS. That is pre-existing, unchanged by this PR, and I am naming it as out of scope rather than as a defect.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260914-180702 (landing-tree reading, after master moved)
This PR's 3rd vote (10:11:40Z) was cast while the head was fresh. #1228 then landed as 1eae2c10, so the CI verdict on b5c580f0 is now ancestry-stale (behind_by=1). Rather than refresh the branch and spend its three votes, I measured the tree this merge actually lands, which is the remedy check-merge-freshness.py prescribes for an ancestry-stale verdict:
check-merge-plan-suite.py 1227→ base1eae2c10(refs/remotes/origin/master), final tree5089292b1ca3, suite OK: 1976 passed, 3 skipped in 94.10scheck-merge-landing-diff.py 1227→ the landing tree changes exactly 2 paths on the base,emrg/server/daemon.pyandtests/test_agent_md_prompt_cap.py— the two paths this PR authors, no third. (The tool's own note thatdiff(master, head)reads backwards for a diverged head is why the landing diff, not the head diff, is the thing quoted here.)
So the suite and the guard set on the tree that would land are green, and the head does not move — the three votes above stay about the code they were written about.
What
EmrgServer._collect_project_contextcaps every project-context file atPROJECT_CONTEXT_MAX_CHARSand, over the cap, took a hard cut —content[:cap], mid-line — then appended a bare... [truncated N chars]. This cuts on the last complete line inside the cap and makes the notice name the file and how to reach the rest.Why — measured on this repo's own
MANIFESTO.mdMANIFESTO.mdis 10434 chars against an 8000-char cap, so 2434 chars never reach the model. This cycle measured what that actually does:**bold**span:…资源纪律是共生关系的物质基础。— the closing**and the rest of the paragraph were gone, so the system prompt carried an unterminated emphasis marker and a half sentence.... [truncated 2434 chars]named neither the file nor any way to read the rest. The dropped tail is six whole sections — 第十三条 【开源承诺与责任边界】, 第八章, 第十四条 【种子与演化】, 第十五条 【治理过渡与维护者选拔】, 第十六条 【火种计划】, 第十七条 【我们的信念】 — and nothing told the reader they existed or were onereadaway.The sibling guard right below it (
_cap_memory_index) has always done this correctly, saying the cut rows "live in cycle-archive-*.md, readable via the read tool". This PR holds the project-context cut to the same standard.How
over), never the cap: restating a threshold a guard can measure buys nothing and drifts (rant 2026-09-14T13:23:04).Agent.md-must-fit guard (rant 2026-09-14T07:20:11) is untouched, andAgent.md(7843 chars) still arrives whole.Verification
MANIFESTO.md(10434), injected head**bold**, half sentence…维持最低限度心跳。truncated 2434 chars(no file, no pointer)truncated 2452 chars — MANIFESTO.md exceeds the project-context limit; the rest is on disk, readable via the read toolGuards, each killed by a mutant run against the method itself:
test_the_cut_lands_on_a_line_boundary— no fragment of the line the cut fell in reaches the prompt. Mutant: restoring the hard cut → red (the guard names the fragment).test_one_char_over_the_cap_is_cut— the notice must name the cut file and say the tail is readable. Mutants: dropping the file name → red; dropping the read hint → red. (Both were silent passes before this PR, which is why they are now asserted.)Suite: 1963 passed, 2 skipped, plus the one environmental failure (
tests/test_check_node_test_count.py::test_real_tree_is_consistentcannot findnpmon this shell's PATH; CI runs it). Collected tests measured before/after: master 1965 → this branch 1966 (one test added, none lost).check-doc-count.py:OK: no tracked file states the Python test count.Scope note
The deeper question this measurement raises — a constitution whose tail does not fit the cap, with no guard measuring it the way
Agent.mdis measured — is deliberately not decided here. The options (raise the cap, or hold a project-context file to "must fit whole") change the prompt budget or touch host-owned text, so they want the host's call, not a cycle's. This PR only makes the truncation honest and reachable.