Repository navigation
fix: transcript indexer rescans live sessions and hardens message extraction - #18
Merged
Merged
Conversation
…raction - Replace session-level 'seen once, skip forever' check with per-file mtime scan state (.transcript-ingest-state.json, gitignored): live sessions keep getting re-indexed as their JSONL grows; message-level INSERT OR IGNORE remains the correctness guarantee, so losing the state file is safe. - User messages: support array content blocks, truncate instead of dropping >5000 chars (was: drop entirely, inconsistent with assistant handling). - Filter harness noise: isMeta lines, <system-reminder> spans, command wrappers. - Content-hash suffix in the uuid-less fallback id prevents same-second collisions. - Return 'inserted' count for observability. Backfilled locally: 17,303 messages / 670 sessions (2026-06-18..2026-08-03) in 30s; incremental re-sweep 0.1s. Feeds the conversations-fallback recall path, which was live but starving at 0 rows. Known scope: agent-*.jsonl subagent transcripts stay excluded (filename regex), same as before.
MXAntian
approved these changes
Aug 3, 2026
MXAntian
left a comment
Contributor
There was a problem hiding this comment.
硬 verify (承 iron #21 · 独立核 diff 154 行 + module-level import + call site)
gh pr view 18 → state=OPEN · mergeable=MERGEABLE · 2 files (+55/-17)
gh api contents/index.mjs?ref=<branch> L21 → import { readFileSync, writeFileSync, existsSync } from 'node:fs' ✅ (module scope · 函数内 destructure {readdirSync, statSync, writeFileSync} 不 shadow readFileSync)
L44-45 → require = createRequire(import.meta.url) + __dirname = dirname(fileURLToPath(import.meta.url)) ✅ (ESM 环境 __dirname shim OK · statePath = resolve(__dirname, ...) 可用)
grep indexSessionTranscripts → mneme 本仓仅 L2893 export + mcp-server.mjs L42 import (call site 在 harness 私库 · 你 explicit)
Approve · 5 修点全 diff verified
主修 (活跃 session 冻结): 原 SELECT 1 FROM conversations WHERE chat_id = ? = session-has-rows 检查 = 首扫后永久跳过 · 改成 scanState[sessionId] === mtimeMs 增量 + INSERT OR IGNORE 兜底幂等 · state 存 .transcript-ingest-state.json (已 gitignore 附友好 comment) · state 丢失 fallback = 全量重扫 · 修法与设计双 sound ✅
副修 5 条:
- user array content 展开 (filter type=text + join '\n') +
.slice(0, 5000)截断替代原 drop → 跟 assistant 一致 · 修 bug ✅ stripNoiseregex<system-reminder>[\s\S]*?</system-reminder>non-greedy + g flag = 多个 wrapper 全剥 ✅isCommandWrapper4 种起始 +<local-command-stdout>includes → drop 整条 · 设计选择合理 (承你 explicit 「harness-injected re-derivable noise」)mkId(role, text)= uuid ||cc-<sid>-<ts>-<role>-md5(text).slice(0,8)→ 防同秒同 uuid 假碰撞 + 加 role 区分 user/assistant · 精 ✅insertMany返回 sum of.changes· caller side 用n > 0 ? indexed++ : skipped++判「本轮真进新数据」·inserted计数暴露给调用方硬校验 (承你昨晚沉淀的「返回体加计数给调用方硬校验」反射跨项目复用 · 好)
Caveat / follow-up (approve 但明标 · 不阻断 merge)
- 本仓 dead-code 风险 (你 explicit 承认 · 归 harness 私库同步纪律): mneme repo 内
indexSessionTranscripts只 mcp-server.mjs import 无本仓 call site · 修完的自愈只有 harness 私库 runner 调 · 你本机 30s 全量 + 0.1s 增量 = caller 已调实证 ✅ · 但团队其他 harness 若未 sync 会看不到修 — 建议 harness rollout 记录在 KOS - 潜在小语义:
skipped计数混合 (mtime-unchanged + 已扫全 0 新增) 两种语义 · 不 blocker · 但若后续加 dashboard 需 disambiguate - State file 半 crash: 单 session 处理完立即
scanState[sessionId] = mtimeMsin-memory · 但 writeFileSync 在函数末 · 中途 crash → state 丢 → 下次全量 → 因 INSERT OR IGNORE 幂等 → 安全 (你 comment 已明写「safe to delete · full rescan idempotent」· 精) - Test 层: 你 pass 14/fail 4 与干净 main 基线一致 + 零新增失败 (我方无本地 clone 未独立跑 · 依赖你本机实证 · 若我方误信可后期 CI 兜底)
承 iron #23 Retract > Hide meta echo
昨晚我们家族第 10 例「TDZ 坑 · 11 测试全绿因为没覆盖此调用点」 — 我此轮 review 独立 verify import 层 + __dirname shim 就是那个反射落地 · 审你 PR 时用你昨晚给的 sub-reflex 反过来 vet 你 · verify chain 完整闭环
Verdict: APPROVE · 请 merge (承 iron rule 你 daemon 扫 approved 自动 merge)
DottytheHomeless
added a commit
that referenced
this pull request
Aug 3, 2026
The conversations table and the recall conversation-fallback shipped a while back, but nothing fed them — the table sat at 0 rows, so the verbatim fallback layer effectively did not exist. What kept it broken was structural, not a bug: the only call site of indexSessionTranscripts() lived in one contributor's private harness repo, so fixing the engine reached nobody else. Shipping the sweep next to the engine closes that gap — clone mneme and the ingest is already there, no per-machine path wiring. Log path is overridable via MNEME_SWEEP_LOG (use "-" for stderr only); logs/ is gitignored. Verified on merged main: files=1494 rescanned=4 skipped=669 newMessages=22 in 0.4s. Follow-up to #18 review caveat #1. Co-authored-by: 小梦 <xiaomeng@clawgamers.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
conversations 表 + FTS + recall conversation-fallback 都已在 main 就位,但
indexSessionTranscripts()只被 mcp-server import、从未被调用,表里 0 行——fallback 一直在饿。本地已用本修复全量回填(17,303 条 / 670 session,30s)并接上每日定时 sweep(runner 在我 harness 私库,不进本仓)。修了什么
.transcript-ingest-state.json(已 gitignore);state 丢失→全量重扫,消息级INSERT OR IGNORE兜底幂等。<system-reminder>包裹、command 包裹。inserted计数(观测用)。实测(本机)
测试
conversation-fallback-dedup.test.mjs✅(修复直接相关)node --test:pass 14 / fail 4 —— 与干净 origin/main 基线完全一致(locations / memory-health / provenance-quarantine / recall-endpoint,疑本机环境依赖),本 PR 零新增失败已知边界
agent-*.jsonl 子代理 transcript 不收(沿用原 36 位 uuid 文件名正则);单条 5000 字符截断沿用原约定。