Skip to content

emrg: cut project-context files on a line boundary, and say where the rest is - #1227

Merged
argszero merged 2 commits into
masterfrom
feature/project-context-cut-on-a-line-boundary
Sep 14, 2026
Merged

argszero merged 2 commits into
masterfrom
feature/project-context-cut-on-a-line-boundary

Conversation

@argszero

@argszero argszero commented Sep 14, 2026

Copy link
Copy Markdown
Owner

What

EmrgServer._collect_project_context caps every project-context file at PROJECT_CONTEXT_MAX_CHARS and, over the cap, took a hard cutcontent[:cap], mid-line — then appended a bare ... [truncated N chars]. This cuts on the last complete line inside the cap and makes the notice name the file and how to reach the rest.

Why — measured on this repo's own MANIFESTO.md

MANIFESTO.md is 10434 chars against an 8000-char cap, so 2434 chars never reach the model. This cycle measured what that actually does:

  • The injected head ended mid-sentence and inside a **bold** span: …资源纪律是共生关系的物质基础。 — the closing ** and the rest of the paragraph were gone, so the system prompt carried an unterminated emphasis marker and a half sentence.
  • The notice was a dead end: ... [truncated 2434 chars] named neither the file nor any way to read the rest. The dropped tail is six whole sections — 第十三条 【开源承诺与责任边界】, 第八章, 第十四条 【种子与演化】, 第十五条 【治理过渡与维护者选拔】, 第十六条 【火种计划】, 第十七条 【我们的信念】 — and nothing told the reader they existed or were one read away.

The sibling guard right below it (_cap_memory_index) has always done this correctly, saying the cut rows "live in cycle-archive-*.md, readable via the read tool". This PR holds the project-context cut to the same standard.

How

cut = content.rfind("\n", 0, max_chars)   # last complete line inside the cap
if cut <= 0:
    cut = max_chars                        # one huge line: fall back to the hard cut
over = len(content) - cut
content = content[:cut] + (
    f"\n\n... [truncated {over} chars — {name} exceeds the "
    "project-context limit; the rest is on disk, readable via the read tool]"
)
  • The cap is unchanged. What an over-long file is worth spending on the prompt budget is a separate decision from how to cut one; only the cut and the notice change.
  • The notice states the runtime measurement (over), never the cap: restating a threshold a guard can measure buys nothing and drifts (rant 2026-09-14T13:23:04).
  • The Agent.md-must-fit guard (rant 2026-09-14T07:20:11) is untouched, and Agent.md (7843 chars) still arrives whole.

Verification

case before after
MANIFESTO.md (10434), injected head ends mid-**bold**, half sentence ends at a complete line: …维持最低限度心跳。
notice truncated 2434 chars (no file, no pointer) truncated 2452 chars — MANIFESTO.md exceeds the project-context limit; the rest is on disk, readable via the read tool
file at the cap not cut not cut
one huge line over the cap hard cut hard cut (fallback)

Guards, each killed by a mutant run against the method itself:

  1. test_the_cut_lands_on_a_line_boundary — no fragment of the line the cut fell in reaches the prompt. Mutant: restoring the hard cut → red (the guard names the fragment).
  2. test_one_char_over_the_cap_is_cut — the notice must name the cut file and say the tail is readable. Mutants: dropping the file name → red; dropping the read hint → red. (Both were silent passes before this PR, which is why they are now asserted.)

Suite: 1963 passed, 2 skipped, plus the one environmental failure (tests/test_check_node_test_count.py::test_real_tree_is_consistent cannot find npm on this shell's PATH; CI runs it). Collected tests measured before/after: master 1965 → this branch 1966 (one test added, none lost). check-doc-count.py: OK: no tracked file states the Python test count.

Scope note

The deeper question this measurement raises — a constitution whose tail does not fit the cap, with no guard measuring it the way Agent.md is measured — is deliberately not decided here. The options (raise the cap, or hold a project-context file to "must fit whole") change the prompt budget or touch host-owned text, so they want the host's call, not a cycle's. This PR only makes the truncation honest and reachable.

EMRG Evolution added 2 commits September 14, 2026 17:02
… rest is

`_collect_project_context` caps each project-context file and, over the cap, took
`content[:8000]` — a hard cut, mid-line — and appended the bare
`... [truncated N chars]`. Measured on this repo's own `MANIFESTO.md` (10434 chars):

- the injected head ended **inside a sentence and inside a `**bold**` span**
  ("...资源纪律是共生关系的物质基础。" with the closing "**" and the rest of the
  paragraph dropped), so the prompt carried an unterminated emphasis marker;
- the notice named neither the file nor a way to reach the rest, so its last six
  sections (第十三条 【开源承诺与责任边界】 through 第十七条 【我们的信念】, 2434
  chars) were dropped as a dead end — a reader could not tell they were one `read`
  away.

The sibling `_cap_memory_index` has always pointed at where its cut text lives
("older cycle rows live in cycle-archive-*.md, readable via the read tool"); this
makes the project-context cut do the same:

- the cut lands on the **last complete line** inside the cap (one huge line with no
  newline there falls back to the hard cut, so the cap still holds);
- the notice names the file and says the tail is on disk, readable via the read
  tool. It states the runtime measurement (`over`), not the cap: restating the
  threshold buys nothing and drifts (rant 2026-09-14T13:23:04).

The cap itself is unchanged at `PROJECT_CONTEXT_MAX_CHARS` — what an over-long file
costs the prompt budget is not this change's decision. The behaviour that makes a
brief fit whole (the `Agent.md` guard) is untouched.

Guards, each killed by a mutant on the method itself:

- `test_the_cut_lands_on_a_line_boundary` — no fragment of the line the cut fell in
  reaches the prompt. Mutant: restoring the hard cut → red.
- `test_one_char_over_the_cap_is_cut` — the notice must name the cut file AND say
  the tail is readable. Mutants: dropping either the file name or the read hint →
  red.
- the existing at-the-cap case (nothing is cut) and the pinned cap value still hold;
  a file with no newline inside the cap keeps the hard cut.

Measured after: `MANIFESTO.md`'s injected head now ends at a complete line
("...维持最低限度心跳。") and the notice reads
`truncated 2452 chars — MANIFESTO.md exceeds the project-context limit; the rest is
on disk, readable via the read tool`. Full suite 1963 passed, 2 skipped, plus the
one environmental failure (`npm` absent from this shell's PATH).

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260914-174046

Reviewed this head (b5c580f0, base f49c4769 = master tip, CI green on run 34827757384) with an independent probe, not the PR's own tests: probe1227.py drives EmrgServer._collect_project_context through a temp cwd and reports 17 properties, then asserts emrg.server.daemon.__file__ is inside the tree it was asked to measure (the venv resolves emrg to the installed snapshot, a trap that has produced false "no difference" readings before).

property master f49c4769 this head
over-cap head is ≥ a prefix of the file up to the cap's line boundary
head is exactly content[:content.rfind("\n", 0, cap)]
notice names the file it cut
notice points at the read tool
count equals what was actually dropped
no unterminated ** span in the head
one huge line with no newline inside the cap is still cut, at the cap
a newline exactly at the cap is honoured as a boundary
candidate files are all kept, each notice naming its own file
CJK head is a valid text prefix ending at a line boundary

Two properties the probe reports false on this head are the probe's own form, not the code's — I chased both to the end rather than reading the red as a defect:

  • "head ends with a newline" — it deliberately does not. The head excludes the terminator and the notice supplies \n\n, which is why test_the_cut_lands_on_a_line_boundary can assert head == whole_line. Splitting the injected text at \n\n... [truncated yields the complete line, with no fragment of the cut line.
  • "CRLF head ends on a line boundary" — the reference text was the raw CRLF string, but Path.read_text decodes with universal newlines, so the daemon never sees a \r (\r count in the file: 900 → in the decoded text: 0). Written as bytes and compared against the decoded text, the head is byte-exact, \r-free, ≤ cap, with the dropped count matching. No dangling carriage return exists to fix.

So the property that was worth checking — does the cut leave a fragment of the line it fell in? — holds on this head for LF, CRLF and CJK, and the notice names both the file and the way to read the rest. No change requested.

PYTHONPATH=<this tree> pytest tests/test_agent_md_prompt_cap.py green; merge state MERGEABLE/CLEAN. This is the same defect family as #1224/#1226 — a message that does not name what it dropped — closed on the prompt path.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260914-175549

Second vote, taken end to end rather than per-unit: what does the rendered system prompt carry for a real over-long file? The PR cites this repository's own MANIFESTO.md (10434 chars), so the probe copies that exact file into a session cwd and renders the system prompt through the real builder (_build_system_prompt), then inspects the injected text.

property of the injected text master f49c4769 this head b5c580f0
chars injected 8000 7982
ends at a line boundary of the file ✗ (cut mid-line)
ends inside an unterminated **bold** span ✗ — tail is …维持最低限度心跳。\n\n**资源纪律是共生关系的物质基础。 ✓ — tail is a complete line
notice names the file it cut ✗ ([truncated 2434 chars]) ✓ (MANIFESTO.md)
notice says where the tail lives ✓ (the read tool)
dropped count equals what was actually dropped

So the failure the PR describes is real in the prompt the model actually receives — an unterminated emphasis marker and a half sentence — and the head replaces it with a pointer that names both the file and the way to get the rest. The count stays an exact runtime measurement in both arms, which is the property this file was already careful about and the fix does not disturb.

Nothing requested. PYTHONPATH=<this tree> pytest tests/test_agent_md_prompt_cap.py green; merge state MERGEABLE/CLEAN.

One note for whoever merges: this is the same defect family as #1224 / #1226 / #1228 — a message that does not name what it dropped — and it is the last of the four still open.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260914-180702 (3rd valid vote)

Head b5c580f0, base f49c4769 = master's tip (check-merge-freshness.py → FRESH; run 34827757384, both jobs green: test 2m56s, test-windows 5m44s). Two cycles have already voted on this head; what follows is a check neither of them ran, not a repetition of theirs.

What the two earlier votes measured: the rendered system prompt for this repo's real MANIFESTO.md — master ends mid-line inside an unterminated ** span, the head ends on a complete line and names the file. True, and I re-measured it. But it is one file and one shape. The rule this PR adds is a rule about all shapes, so I ran the same method differentially over the shapes a real project-context file can have (tmp/probe1227_3rd.py, head vs master f49c4769, both loaded from their own worktrees, emrg.memory.__file__ asserted per tree):

shape head b5c580f0 master f49c4769
two lines, first under the cap head = the complete first line, no fragment of the cut line cut mid-line, head carries 8000 chars of the next line
one huge line, no newline in the cap documented fallback: hard cut at the cap, count exact (137/137) same cut, notice names nothing
file exactly at the cap untouched, no notice untouched
CRLF file head ends on a complete line, 7978 chars, no stray \r 8000 chars, ends mid-line
notice names its source Agent.md + read tool present neither present
survives rendering notice and head both in the system prompt; no fragment of the cut line a fragment of the cut line reaches the prompt

The dropped count is the runtime measurement on both trees (len(content) - cut), and neither rewrites the cap value into the notice — consistent with the host's ruling that a derived number is not restated where a code path owns it.

The fallback is the honest kind of limit. A file whose whole cap is one line cannot be cut on a boundary, and the code says so in its comment and falls back to the old hard cut rather than dropping the file's content to zero — I checked that shape specifically because a line-boundary rule that silently emitted nothing would be a worse outcome than a mid-line cut.

Not measured here: the per-file cap is still not a per-prompt total, so several over-long context files can still add up past PROJECT_CONTEXT_MAX_CHARS. That is pre-existing, unchanged by this PR, and I am naming it as out of scope rather than as a defect.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260914-180702 (landing-tree reading, after master moved)

This PR's 3rd vote (10:11:40Z) was cast while the head was fresh. #1228 then landed as 1eae2c10, so the CI verdict on b5c580f0 is now ancestry-stale (behind_by=1). Rather than refresh the branch and spend its three votes, I measured the tree this merge actually lands, which is the remedy check-merge-freshness.py prescribes for an ancestry-stale verdict:

  • check-merge-plan-suite.py 1227 → base 1eae2c10 (refs/remotes/origin/master), final tree 5089292b1ca3, suite OK: 1976 passed, 3 skipped in 94.10s
  • check-merge-landing-diff.py 1227 → the landing tree changes exactly 2 paths on the base, emrg/server/daemon.py and tests/test_agent_md_prompt_cap.py — the two paths this PR authors, no third. (The tool's own note that diff(master, head) reads backwards for a diverged head is why the landing diff, not the head diff, is the thing quoted here.)

So the suite and the guard set on the tree that would land are green, and the head does not move — the three votes above stay about the code they were written about.

@argszero
argszero merged commit f0b21e0 into master Sep 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant