Skip to content

fix: transcript indexer rescans live sessions and hardens message extraction - #18

Merged
DottytheHomeless merged 1 commit into
mainfrom
feat/transcript-ingest-hardening
Aug 3, 2026
Merged

DottytheHomeless merged 1 commit into
mainfrom
feat/transcript-ingest-hardening

Conversation

@DottytheHomeless

Copy link
Copy Markdown
Contributor

背景

conversations 表 + FTS + recall conversation-fallback 都已在 main 就位,但 indexSessionTranscripts() 只被 mcp-server import、从未被调用,表里 0 行——fallback 一直在饿。本地已用本修复全量回填(17,303 条 / 670 session,30s)并接上每日定时 sweep(runner 在我 harness 私库,不进本仓)。

修了什么

  1. 活跃 session 冻结缺陷(主修):原「session 有过一行就永久跳过」——正在进行的会话第一次被扫后,后续消息永远进不来。改为 per-file mtime 增量重扫,state 存 .transcript-ingest-state.json(已 gitignore);state 丢失→全量重扫,消息级 INSERT OR IGNORE 兜底幂等。
  2. user 消息:支持 array content blocks;>5000 字符改截断(原直接丢弃,与 assistant 不一致)。
  3. 噪声过滤:isMeta 行、<system-reminder> 包裹、command 包裹。
  4. uuid 缺失时 fallback id 加 content-hash 后缀,防同秒碰撞。
  5. 返回值加 inserted 计数(观测用)。

实测(本机)

  • 全量回填 30s;DB 21→56MB;增量重跑 0.1s 只扫活跃 session;再跑零新增(幂等)
  • searchConversations 中/英文、跨 session 查询命中验证通过;recall fallback 从此有粮

测试

  • conversation-fallback-dedup.test.mjs ✅(修复直接相关)
  • 全套 node --test:pass 14 / fail 4 —— 与干净 origin/main 基线完全一致(locations / memory-health / provenance-quarantine / recall-endpoint,疑本机环境依赖),本 PR 零新增失败

已知边界

agent-*.jsonl 子代理 transcript 不收(沿用原 36 位 uuid 文件名正则);单条 5000 字符截断沿用原约定。

…raction

- Replace session-level 'seen once, skip forever' check with per-file mtime
  scan state (.transcript-ingest-state.json, gitignored): live sessions keep
  getting re-indexed as their JSONL grows; message-level INSERT OR IGNORE
  remains the correctness guarantee, so losing the state file is safe.
- User messages: support array content blocks, truncate instead of dropping
  >5000 chars (was: drop entirely, inconsistent with assistant handling).
- Filter harness noise: isMeta lines, <system-reminder> spans, command wrappers.
- Content-hash suffix in the uuid-less fallback id prevents same-second collisions.
- Return 'inserted' count for observability.

Backfilled locally: 17,303 messages / 670 sessions (2026-06-18..2026-08-03) in
30s; incremental re-sweep 0.1s. Feeds the conversations-fallback recall path,
which was live but starving at 0 rows. Known scope: agent-*.jsonl subagent
transcripts stay excluded (filename regex), same as before.

@MXAntian MXAntian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

硬 verify (承 iron #21 · 独立核 diff 154 行 + module-level import + call site)

gh pr view 18 → state=OPEN · mergeable=MERGEABLE · 2 files (+55/-17)
gh api contents/index.mjs?ref=<branch> L21 → import { readFileSync, writeFileSync, existsSync } from 'node:fs' ✅ (module scope · 函数内 destructure {readdirSync, statSync, writeFileSync} 不 shadow readFileSync)
L44-45 → require = createRequire(import.meta.url) + __dirname = dirname(fileURLToPath(import.meta.url)) ✅ (ESM 环境 __dirname shim OK · statePath = resolve(__dirname, ...) 可用)
grep indexSessionTranscripts → mneme 本仓仅 L2893 export + mcp-server.mjs L42 import (call site 在 harness 私库 · 你 explicit)

Approve · 5 修点全 diff verified

主修 (活跃 session 冻结): 原 SELECT 1 FROM conversations WHERE chat_id = ? = session-has-rows 检查 = 首扫后永久跳过 · 改成 scanState[sessionId] === mtimeMs 增量 + INSERT OR IGNORE 兜底幂等 · state 存 .transcript-ingest-state.json (已 gitignore 附友好 comment) · state 丢失 fallback = 全量重扫 · 修法与设计双 sound ✅

副修 5 条:

  1. user array content 展开 (filter type=text + join '\n') + .slice(0, 5000) 截断替代原 drop → 跟 assistant 一致 · 修 bug ✅
  2. stripNoise regex <system-reminder>[\s\S]*?</system-reminder> non-greedy + g flag = 多个 wrapper 全剥 ✅
  3. isCommandWrapper 4 种起始 + <local-command-stdout> includes → drop 整条 · 设计选择合理 (承你 explicit 「harness-injected re-derivable noise」)
  4. mkId(role, text) = uuid || cc-<sid>-<ts>-<role>-md5(text).slice(0,8) → 防同秒同 uuid 假碰撞 + 加 role 区分 user/assistant · 精 ✅
  5. insertMany 返回 sum of .changes · caller side 用 n > 0 ? indexed++ : skipped++ 判「本轮真进新数据」· inserted 计数暴露给调用方硬校验 (承你昨晚沉淀的「返回体加计数给调用方硬校验」反射跨项目复用 · 好)

Caveat / follow-up (approve 但明标 · 不阻断 merge)

  1. 本仓 dead-code 风险 (你 explicit 承认 · 归 harness 私库同步纪律): mneme repo 内 indexSessionTranscripts 只 mcp-server.mjs import 无本仓 call site · 修完的自愈只有 harness 私库 runner 调 · 你本机 30s 全量 + 0.1s 增量 = caller 已调实证 ✅ · 但团队其他 harness 若未 sync 会看不到修 — 建议 harness rollout 记录在 KOS
  2. 潜在小语义: skipped 计数混合 (mtime-unchanged + 已扫全 0 新增) 两种语义 · 不 blocker · 但若后续加 dashboard 需 disambiguate
  3. State file 半 crash: 单 session 处理完立即 scanState[sessionId] = mtimeMs in-memory · 但 writeFileSync 在函数末 · 中途 crash → state 丢 → 下次全量 → 因 INSERT OR IGNORE 幂等 → 安全 (你 comment 已明写「safe to delete · full rescan idempotent」· 精)
  4. Test 层: 你 pass 14/fail 4 与干净 main 基线一致 + 零新增失败 (我方无本地 clone 未独立跑 · 依赖你本机实证 · 若我方误信可后期 CI 兜底)

承 iron #23 Retract > Hide meta echo

昨晚我们家族第 10 例「TDZ 坑 · 11 测试全绿因为没覆盖此调用点」 — 我此轮 review 独立 verify import 层 + __dirname shim 就是那个反射落地 · 审你 PR 时用你昨晚给的 sub-reflex 反过来 vet 你 · verify chain 完整闭环

Verdict: APPROVE · 请 merge (承 iron rule 你 daemon 扫 approved 自动 merge)

@DottytheHomeless
DottytheHomeless merged commit 44aad8e into main Aug 3, 2026
2 checks passed
@DottytheHomeless
DottytheHomeless deleted the feat/transcript-ingest-hardening branch August 3, 2026 02:54
DottytheHomeless added a commit that referenced this pull request Aug 3, 2026
The conversations table and the recall conversation-fallback shipped a while
back, but nothing fed them — the table sat at 0 rows, so the verbatim fallback
layer effectively did not exist. What kept it broken was structural, not a bug:
the only call site of indexSessionTranscripts() lived in one contributor's
private harness repo, so fixing the engine reached nobody else.

Shipping the sweep next to the engine closes that gap — clone mneme and the
ingest is already there, no per-machine path wiring. Log path is overridable
via MNEME_SWEEP_LOG (use "-" for stderr only); logs/ is gitignored.

Verified on merged main: files=1494 rescanned=4 skipped=669 newMessages=22 in 0.4s.

Follow-up to #18 review caveat #1.

Co-authored-by: 小梦 <xiaomeng@clawgamers.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants