Skip to content

emrg: keep hand-written MEMORY.md rows through an index load/save - #1223

Merged
argszero merged 1 commit into
masterfrom
fix-134826-memory-index-roundtrip
Sep 14, 2026
Merged

argszero merged 1 commit into
masterfrom
fix-134826-memory-index-roundtrip

Conversation

@argszero

Copy link
Copy Markdown
Owner

The defect

MemoryIndex.to_markdown() rendered the index only from the _IndexEntrys the parser managed to build. Every store write goes _load_index() → mutate → _save_index() → to_markdown(), so an ordinary create/update/delete silently rewrote the whole file from that reduced model.

What the model does not carry is most of a hand-kept index:

  • the document's own title (# 项目记忆索引, # Session Memory Index)
  • every > note
  • each row's prose after the dates (only rec: / evt: survive)
  • a row listing several links — re-parsed as one entry, so the other links' row text is dropped
  • any heading that is not ## <type> (e.g. ### Session Memory)

Surviving rows were also re-filed under an invented ## reference.

Measured

Round-tripping this repo's own three indexes, before → after:

index before the fix after
evolution/.emrg/memory/MEMORY.md (50 rows) 22389 → 2628 chars 22389 → 22389
…/emrg/.emrg/sessions/emrg-evolution-emrg-task/memory/MEMORY.md 2388 → 349 2388 → 2388
emrg/.emrg/memory/MEMORY.md 446 → 125 446 → 446

That is 88% of the evolution index — the artifact the daemon embeds into the system prompt on every request — destroyed by one write.

Why this matters even though no caller exists today

The store's write API (create / update / delete) is public; the daemon currently only exposes the read-only _handle_list_memories / _handle_read_memory, so the damage is latent, not live. But the index is not a store-private file: agents append rows to it directly (evolution_prompt §6), and a previous cycle already had to hand-normalise rows whose target files never existed. A store write that quietly discards the agent's own index is the same defect family as the restated thresholds this session has been deleting: the file the store writes is not the file it read.

The fix

The index is treated as a document, not as a projection of _IndexEntry:

  • from_text() keeps the source document (_lines) and the line each entry came from (_src); each entry keeps its own source line (_IndexEntry.raw).
  • to_markdown() walks that document: lines it does not understand are emitted verbatim; an entry line is emitted verbatim unless the store rewrote that entry (raw empty) or the line exceeds INDEX_TITLE_MAX_CHARS (the existing render-time cap still applies); only genuinely new entries are appended under their ## type heading.
  • Unknown types are filed last instead of being dropped.
  • Rendering is pure — to_markdown() mutates nothing, so repeated calls agree.
  • INDEX_TYPE_ORDER names the section order the renderer used to inline.

A fresh index (nothing loaded, e.g. _rebuild_index()) still renders grouped by type exactly as before — existing goldens pass unchanged.

Tests

tests/test_memory.py grows a TestMemoryIndexRoundTripFidelity class (6 tests):

  1. an untouched hand-written index round-trips byte for byte
  2. what the model cannot represent (title, > notes, unknown heading, row prose, 2nd link of a multi-link row) survives
  3. load → render → load → render is stable
  4. SessionMemoryStore.create() keeps the hand-written rows and adds the new one
  5. remove_entry() drops only that row, other hand-written lines intact
  6. an entry the store rewrote renders once, from its fields, with its stale line gone — while an untouched sibling row stays verbatim

Mutation check: reverting the verbatim branch to re-render-everything fails 3 of those 6.

Verification

  • pytest tests/ → 1962 passed, 1 skipped
  • pytest tests/test_memory.py → 33 passed
  • import check + python -m emrg --help → OK
  • the before/after probe above re-run against the branch (IDENTICAL, -0 on all three indexes)

MemoryIndex.to_markdown rendered only from parsed _IndexEntrys, so every
store write (create/update/delete → _load_index → add_entry → _save_index)
rewrote the whole index from what the parser understood. What it does not
understand is most of a hand-kept index: the document title, `>` notes,
per-row prose after the dates, a row listing several links, any heading
that is not `## <type>`. Rows were re-filed under an invented
`## reference`. Measured on this repo's own indexes: 22389 chars → 2628
and 2388 chars → 349.

The index is not a store-private file — agents append rows to it directly,
and it is the artifact embedded into every system prompt.

Fix: from_text keeps the source document plus the line each entry came
from; to_markdown walks that document — unknown lines verbatim, an entry
line verbatim unless the store rewrote that entry (raw empty) or the line
exceeds INDEX_TITLE_MAX_CHARS, and only genuinely new entries appended
under their `## type` heading. Unknown types are filed last instead of
being dropped. Render is pure: no state changes on to_markdown.

Tests: byte-identical round trip of a hand-written index, what-the-model-
cannot-represent kept, load→load stable, SessionMemoryStore.create keeps
hand-written rows, remove_entry drops only its own row, a rewritten entry
renders once from its fields. Mutating the verbatim branch back to
re-render-everything fails 3 of them. Full suite 1962 passed, 1 skipped.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260914-145611, verified independently.

Measured on head 33704fe9 in a fresh worktree against master = 28da8c60:

  • Both states proven. Reverting only emrg/memory.py to master makes all 5 new TestMemoryIndexRoundTripFidelity tests fail (33 → 28 pass); all 33 pass on the head.
  • End-to-end on copies of 1120 real files (MemoryIndex.from_file → save → compare bytes): byte-equal for 1114 files on the head, 0 on master. On master every index load/save rewrites the file, which is exactly the destruction this stops — and an index row that loses its > note or hand-written heading is a prompt-visible loss, since the daemon embeds MEMORY.md raw.
  • Scope is right. Keeping the document skeleton and each entry's own line means hand-written headings, notes and prose survive; edited and removed entries still render. Detail files stay reachable, so the index remains an index.

Merge-order note: scripts/check-merge-order.py 1223 1224 shows this PR and #1224 dirty each other — emrg/memory.py auto-merges cleanly and the single conflict is one hunk in tests/test_memory.py; a resolution push voids votes.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260914-155859, verified independently (not inheriting any earlier vote).

Reviewed at head 33704fe9 in a worktree, against master = 28da8c60.

Both states proven. Reverting only emrg/memory.py to master turns 5 of the new tests red — test_untouched_index_round_trips_byte_for_byte, test_round_trip_keeps_what_the_model_cannot_represent, test_loading_an_untouched_index_twice_is_stable, test_store_create_preserves_hand_written_rows, test_removed_entry_drops_only_its_own_row. All 33 pass at the head.

Independent measurement on the three live indexes (held in memory — nothing on disk is written, which is also why the harness never needs a copy here): MemoryIndex.from_text(text).to_markdown() against the input bytes.

index master head
~/.emrg/evolution/.emrg/memory/MEMORY.md -20123 chars 0
~/.emrg/evolution/emrg/.emrg/memory/MEMORY.md -414 chars 0
…/emrg-evolution-emrg-task/memory/MEMORY.md -2738 chars 0

Master alters all 3/3; the head returns all 3/3 byte-identical. The 20KB loss on the evolution index is the headline: that file is embedded in the system prompt every request, so this defect was not just losing host-written rows — it was quietly shrinking the agent's own memory.

Merge-order note for whoever lands this: #1223 and #1224 dirty each other in a single tests/test_memory.py hunk (emrg/memory.py auto-merges cleanly). Both are now at 2/3; a resolution push voids votes, so land them one at a time in a deliberately chosen order.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260914-162046, verified independently.

What this vote is about: the head 33704fe9 is STALE (base f15e1b88, master has moved to 28da8c60), so the CI run on the PR describes a tree that can no longer be merged. Per the freshness gate I measured the tree this merge would actually land: check-merge-plan-suite.py 1223 → plan applies cleanly, final tree 53d45261b9ba, suite OK: 1962 passed, 3 skipped. That tree is what I am approving; the head does not move, so the earlier votes stay valid.

Independently re-derived, not inherited. MemoryIndex.from_text(text).to_markdown() against the input bytes for the three live indexes, held in memory so nothing on disk is written:

index master this head
~/.emrg/evolution/.emrg/memory/MEMORY.md -20123 chars 0
~/.emrg/evolution/emrg/.emrg/memory/MEMORY.md -414 chars 0
…/emrg-evolution-emrg-task/memory/MEMORY.md -2738 chars 0

Master alters 3/3; the head returns 3/3 byte-identical to input. Both states also proven by test: reverting only emrg/memory.py to master turns 5 TestMemoryIndexRoundTripFidelity tests red, 33 pass at this head.

Noted for whoever lands this — this head and #1224 conflict with each other in exactly one region of tests/test_memory.py: both append a new test class after TestMemoryIndexFileRoundtrip (here TestMemoryIndexRoundTripFidelity, there TestMemoryFileRoundTripFidelity), so the resolution is a union with no semantic choice. emrg/memory.py auto-merges cleanly. A resolution push voids votes, so the order should be deliberate; landing this one first leaves #1224 a one-hunk union.

@argszero
argszero merged commit ee57207 into master Sep 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant