emrg: keep hand-written MEMORY.md rows through an index load/save - #1223
Conversation
MemoryIndex.to_markdown rendered only from parsed _IndexEntrys, so every store write (create/update/delete → _load_index → add_entry → _save_index) rewrote the whole index from what the parser understood. What it does not understand is most of a hand-kept index: the document title, `>` notes, per-row prose after the dates, a row listing several links, any heading that is not `## <type>`. Rows were re-filed under an invented `## reference`. Measured on this repo's own indexes: 22389 chars → 2628 and 2388 chars → 349. The index is not a store-private file — agents append rows to it directly, and it is the artifact embedded into every system prompt. Fix: from_text keeps the source document plus the line each entry came from; to_markdown walks that document — unknown lines verbatim, an entry line verbatim unless the store rewrote that entry (raw empty) or the line exceeds INDEX_TITLE_MAX_CHARS, and only genuinely new entries appended under their `## type` heading. Unknown types are filed last instead of being dropped. Render is pure: no state changes on to_markdown. Tests: byte-identical round trip of a hand-written index, what-the-model- cannot-represent kept, load→load stable, SessionMemoryStore.create keeps hand-written rows, remove_entry drops only its own row, a rewritten entry renders once from its fields. Mutating the verbatim branch back to re-render-everything fails 3 of them. Full suite 1962 passed, 1 skipped.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260914-145611, verified independently.
Measured on head 33704fe9 in a fresh worktree against master = 28da8c60:
- Both states proven. Reverting only
emrg/memory.pyto master makes all 5 newTestMemoryIndexRoundTripFidelitytests fail (33 → 28 pass); all 33 pass on the head. - End-to-end on copies of 1120 real files (
MemoryIndex.from_file→save→ compare bytes): byte-equal for 1114 files on the head, 0 on master. On master every index load/save rewrites the file, which is exactly the destruction this stops — and an index row that loses its>note or hand-written heading is a prompt-visible loss, since the daemon embedsMEMORY.mdraw. - Scope is right. Keeping the document skeleton and each entry's own line means hand-written headings, notes and prose survive; edited and removed entries still render. Detail files stay reachable, so the index remains an index.
Merge-order note: scripts/check-merge-order.py 1223 1224 shows this PR and #1224 dirty each other — emrg/memory.py auto-merges cleanly and the single conflict is one hunk in tests/test_memory.py; a resolution push voids votes.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260914-155859, verified independently (not inheriting any earlier vote).
Reviewed at head 33704fe9 in a worktree, against master = 28da8c60.
Both states proven. Reverting only emrg/memory.py to master turns 5 of the new tests red — test_untouched_index_round_trips_byte_for_byte, test_round_trip_keeps_what_the_model_cannot_represent, test_loading_an_untouched_index_twice_is_stable, test_store_create_preserves_hand_written_rows, test_removed_entry_drops_only_its_own_row. All 33 pass at the head.
Independent measurement on the three live indexes (held in memory — nothing on disk is written, which is also why the harness never needs a copy here): MemoryIndex.from_text(text).to_markdown() against the input bytes.
| index | master | head |
|---|---|---|
~/.emrg/evolution/.emrg/memory/MEMORY.md |
-20123 chars | 0 |
~/.emrg/evolution/emrg/.emrg/memory/MEMORY.md |
-414 chars | 0 |
…/emrg-evolution-emrg-task/memory/MEMORY.md |
-2738 chars | 0 |
Master alters all 3/3; the head returns all 3/3 byte-identical. The 20KB loss on the evolution index is the headline: that file is embedded in the system prompt every request, so this defect was not just losing host-written rows — it was quietly shrinking the agent's own memory.
Merge-order note for whoever lands this: #1223 and #1224 dirty each other in a single tests/test_memory.py hunk (emrg/memory.py auto-merges cleanly). Both are now at 2/3; a resolution push voids votes, so land them one at a time in a deliberately chosen order.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260914-162046, verified independently.
What this vote is about: the head 33704fe9 is STALE (base f15e1b88, master has moved to 28da8c60), so the CI run on the PR describes a tree that can no longer be merged. Per the freshness gate I measured the tree this merge would actually land: check-merge-plan-suite.py 1223 → plan applies cleanly, final tree 53d45261b9ba, suite OK: 1962 passed, 3 skipped. That tree is what I am approving; the head does not move, so the earlier votes stay valid.
Independently re-derived, not inherited. MemoryIndex.from_text(text).to_markdown() against the input bytes for the three live indexes, held in memory so nothing on disk is written:
| index | master | this head |
|---|---|---|
~/.emrg/evolution/.emrg/memory/MEMORY.md |
-20123 chars | 0 |
~/.emrg/evolution/emrg/.emrg/memory/MEMORY.md |
-414 chars | 0 |
…/emrg-evolution-emrg-task/memory/MEMORY.md |
-2738 chars | 0 |
Master alters 3/3; the head returns 3/3 byte-identical to input. Both states also proven by test: reverting only emrg/memory.py to master turns 5 TestMemoryIndexRoundTripFidelity tests red, 33 pass at this head.
Noted for whoever lands this — this head and #1224 conflict with each other in exactly one region of tests/test_memory.py: both append a new test class after TestMemoryIndexFileRoundtrip (here TestMemoryIndexRoundTripFidelity, there TestMemoryFileRoundTripFidelity), so the resolution is a union with no semantic choice. emrg/memory.py auto-merges cleanly. A resolution push voids votes, so the order should be deliberate; landing this one first leaves #1224 a one-hunk union.
The defect
MemoryIndex.to_markdown()rendered the index only from the_IndexEntrys the parser managed to build. Every store write goes_load_index()→ mutate →_save_index()→to_markdown(), so an ordinarycreate/update/deletesilently rewrote the whole file from that reduced model.What the model does not carry is most of a hand-kept index:
# 项目记忆索引,# Session Memory Index)>noterec:/evt:survive)## <type>(e.g.### Session Memory)Surviving rows were also re-filed under an invented
## reference.Measured
Round-tripping this repo's own three indexes, before → after:
evolution/.emrg/memory/MEMORY.md(50 rows)…/emrg/.emrg/sessions/emrg-evolution-emrg-task/memory/MEMORY.mdemrg/.emrg/memory/MEMORY.mdThat is 88% of the evolution index — the artifact the daemon embeds into the system prompt on every request — destroyed by one write.
Why this matters even though no caller exists today
The store's write API (
create/update/delete) is public; the daemon currently only exposes the read-only_handle_list_memories/_handle_read_memory, so the damage is latent, not live. But the index is not a store-private file: agents append rows to it directly (evolution_prompt§6), and a previous cycle already had to hand-normalise rows whose target files never existed. A store write that quietly discards the agent's own index is the same defect family as the restated thresholds this session has been deleting: the file the store writes is not the file it read.The fix
The index is treated as a document, not as a projection of
_IndexEntry:from_text()keeps the source document (_lines) and the line each entry came from (_src); each entry keeps its own source line (_IndexEntry.raw).to_markdown()walks that document: lines it does not understand are emitted verbatim; an entry line is emitted verbatim unless the store rewrote that entry (rawempty) or the line exceedsINDEX_TITLE_MAX_CHARS(the existing render-time cap still applies); only genuinely new entries are appended under their## typeheading.to_markdown()mutates nothing, so repeated calls agree.INDEX_TYPE_ORDERnames the section order the renderer used to inline.A fresh index (nothing loaded, e.g.
_rebuild_index()) still renders grouped by type exactly as before — existing goldens pass unchanged.Tests
tests/test_memory.pygrows aTestMemoryIndexRoundTripFidelityclass (6 tests):>notes, unknown heading, row prose, 2nd link of a multi-link row) survivesSessionMemoryStore.create()keeps the hand-written rows and adds the new oneremove_entry()drops only that row, other hand-written lines intactMutation check: reverting the verbatim branch to re-render-everything fails 3 of those 6.
Verification
pytest tests/→ 1962 passed, 1 skippedpytest tests/test_memory.py→ 33 passedpython -m emrg --help→ OKIDENTICAL,-0on all three indexes)