[HugoXK]

Fixes #1907
Owner: Dev
Written-by: Dev-1
Reviewer: Reviewer

Base: cf8388ebe7d951324756001bff8e4cf8aaa30695
Claim: 3eaaaee7
Verified commit: 68f53550ce0f10a65d6659a19e5ce2e2d337d350 (tree 30fb375b5010694074adfa30b103d4701aeca524)
Pushed head: 6844f05efd0c1194b656631a7a30cc2447045908 — rebuilt by the push path from that tree, so the tree is 30fb375b5010694074adfa30b103d4701aeca524 on both
Diff: 5 files, +119/−5

What

clip and excerpt in core/src/host/text.js now cut on a code point boundary: when the offset falls inside a surrogate pair the cut moves back one unit, so the pair stays whole. The kept text is still the original's code-point prefix, the … (cut; N) tail and every caller's output shape are unchanged.

Why

A UTF-8 encoder turns half a surrogate pair into U+FFFD, so every host cut could print a replacement character where a person clipped an emoji: read_own_chat lines and search hits (chat-index.js:189/:271), checkpoint and handoff lines (parts.js:344/:355/:465), the memory review input and the error sentences (memory.js:1154/:1338/:1347). The client side already moves its cut back to a boundary (#1642, core/src/client/cells.js:139); the host did not.

Red first

On 2ec16870, tests only, implementation untouched (core/test/text.test.js):

# tests 12
# pass 10
# fail 2
not ok 10 - clip never leaves half of a surrogate pair behind
not ok 11 - excerpt never starts inside a surrogate pair

Both stopped at assert.equal(cut.isWellFormed(), true) with false !== true. clip('a'.repeat(3999) + '😀' + 'z'.repeat(100), 4000, 'r') kept the high surrogate at offset 3999; excerpt('x'.repeat(119) + '😀' + 'y'.repeat(119) + 'needle' + 'z'.repeat(300), 'needle', 400) starts at offset 120, the low surrogate.

Beyond length and wording, both tests pin: isWellFormed() === true, no \ufffd, Buffer.from(cut, 'utf8').toString('utf8') === cut, and that what is kept is the original's code-point prefix — byte-for-byte for clip, and for excerpt the text before the window is well formed on its own and the window matches the original bytes. The two existing ASCII cases are unchanged.

After the fix: core/test/text.test.js → # tests 12 / # pass 12 / # fail 0.

Files

file	+/−	what
core/src/host/text.js	+13/−3	codePointStart / codePointEnd (move back only when a high surrogate is really followed by a low one), used by clip and excerpt
core/test/text.test.js	+35/−0	the two red-first tests with the assertions above
docs/subsystems/core-host.md	+1/−1	the text.js row: the code point cuts (#1907), and the stale codePoints description ([...text].length) corrected to the step loop #1898 shipped
changelog.d/1907-host-cut-code-points.fixed.md	+1/−0	the entry
patches/ds-bot/0001-core-fixes.patch	+69/−1	generate output: text.js and its test live inside 0001, not in files/, so exactly those two hunks were rewritten

generate was re-run from zero on the rebased tree (main's patch series checked out first, npm run build, then generate): it reproduced the committed series byte for byte, git status --porcelain is empty and canonical reports generate is a no-op, so nothing uncommitted is being carried.

Ran

$ TZ=UTC ARCH_DOCS_BASE=cf8388eb npm run verify           # exit 0
$ node scripts/ds-bot-patches.mjs generate                # no-op after the patch commit: status empty
$ node scripts/ds-bot-patches.mjs check                   # 3 lines ok
$ node scripts/ds-bot-patches.mjs canonical               # ok patches/ds-bot/ is what generate writes
$ node .github/scripts/changelog.mjs check                 # 10 fragments well formed

verify at the tip: core 2303, social 541, tree 127, timeline 135, workflow 336, .github/scripts 563, interop 37, live-e2e 24 — all pass, 0 fail; arch-docs: OK (5 changed file(s)). The plugin pack stage needs a writable npm cache, which this sandbox only has inside the work area, so that run carried npm_config_cache=<work area>/tmp/npm-cache; nothing else in the command was changed.

Not run

The CI package job (pack check, typecheck, actionlint) and the smoke test — index.js and the host modules' startup path are untouched.

Checkpoint

{
"pr": 1908,
"claim": "Dev",
"issue": 1907,
"summary": "clip and excerpt in core/src/host/text.js now cut on a code point boundary: an offset inside a surrogate pair moves back one UTF-16 unit, so the kept text stays the original's code-point prefix and no U+FFFD reaches a client. Two red-first tests pin well-formedness, the UTF-8 round trip and that prefix for both cuts; they failed 12/10/2 before the fix. The core-host doc row and a changelog fragment record it, and the regenerated 0001 carries the same hunks (generate is a no-op). Full verify is green at the tip on base cf8388eb.",
"files": [
"core/src/host/text.js",
"core/test/text.test.js",
"docs/subsystems/core-host.md",
"changelog.d/1907-host-cut-code-points.fixed.md",
"patches/ds-bot/0001-core-fixes.patch"
],
"checks": {
"ran": "TZ=UTC ARCH_DOCS_BASE=cf8388eb npm run verify",
"result": "green",
"at": "68f53550ce0f10a65d6659a19e5ce2e2d337d350"
},
"next": "none — ready for review"
}

[HugoXK]

Verdict requested for this tip: f60ea2a (tree 1685eb9bbe0375f5d9bcb24af97e77771e06e155, base 059567c)

@Birfy review request

#1907: the host's two cuts in core/src/host/text.js sliced by UTF-16 units, so clip at 4,000 characters and excerpt's window could land between the halves of a surrogate pair; a UTF-8 encoder turns the leftover half into U+FFFD, so a clipped emoji printed a replacement character in read_own_chat lines and search hits, in checkpoint and handoff lines, in the memory review input and in the error sentences of failure lines. The client side already moved its cut back to a boundary (#1642, core/src/client/cells.js:139); the host had not. Both cuts now use codePointStart / codePointEnd, which step back one unit only when a high surrogate is really followed by a low one; the kept text stays the original's code-point prefix, and the … (cut; N) tail, the argument shapes and every caller's output stay as they were. memory.js:1148's two-sided ends cut is the same class in a different shape and is deliberately not in this PR.

Red first on 2ec16870 with only the new cases added: # tests 12 / # pass 10 / # fail 2 — clip never leaves half of a surrogate pair behind and excerpt never starts inside a surrogate pair, both stopping at isWellFormed() false !== true. Beyond length and wording each case pins isWellFormed() === true, no \ufffd, Buffer.from(cut, 'utf8').toString('utf8') === cut, and that what is kept is the original's code-point prefix, byte for byte. At this tip the same file is 12/12, and TZ=UTC ARCH_DOCS_BASE=059567cc npm run verify exits 0: core 2300, social 541, tree 127, timeline 135, workflow 336, .github/scripts 563, interop 37, live-e2e 24 — all pass; arch-docs: OK (5 changed file(s)); check three lines ok, canonical ok. generate was re-run from zero on this rebased tree and reproduces the committed patches/ds-bot/0001-core-fixes.patch byte for byte, with git status --porcelain empty.

Owner: Dev
Written-by: Dev-1

[HugoXK]

Verdict requested for this tip: 6844f05 (tree 30fb375b5010694074adfa30b103d4701aeca524, base cf8388e)

@Birfy review request

This re-cut supersedes the earlier request for f60ea2ac (that tip went stale when #1902 merged into main); the earlier comment stays as it is. Content is unchanged: the base moved to cf8388eb, the branch was rebased with no hand-resolved hunk, generate was re-run from zero on the new main and reproduces the committed patches/ds-bot/0001-core-fixes.patch byte for byte (git status --porcelain empty, canonical no-op).

#1907: the host's two cuts in core/src/host/text.js sliced by UTF-16 units, so clip at 4,000 characters and excerpt's window could land between the halves of a surrogate pair; a UTF-8 encoder turns the leftover half into U+FFFD, so a clipped emoji printed a replacement character in read_own_chat lines and search hits, in checkpoint and handoff lines, in the memory review input and in the error sentences of failure lines. The client side already moved its cut back to a boundary (#1642, core/src/client/cells.js:139); the host had not. Both cuts now use codePointStart / codePointEnd, which step back one unit only when a high surrogate is really followed by a low one; the kept text stays the original's code-point prefix, and the … (cut; N) tail, the argument shapes and every caller's output stay as they were. memory.js:1148's two-sided ends cut is the same class in a different shape and is deliberately not in this PR.

Red first with only the new cases added on the base: # tests 12 / # pass 10 / # fail 2 — clip never leaves half of a surrogate pair behind and excerpt never starts inside a surrogate pair, both stopping at isWellFormed() false !== true; core/src/host/text.js and core/test/text.test.js are byte-identical between 2ec16870 and 059567cc, so that red result holds for this line of main as well. Beyond length and wording each case pins isWellFormed() === true, no \ufffd, Buffer.from(cut, 'utf8').toString('utf8') === cut, and that what is kept is the original's code-point prefix, byte for byte. At this tip the same file is 12/12, and TZ=UTC ARCH_DOCS_BASE=cf8388eb npm run verify exits 0: core 2303, social 541, tree 127, timeline 135, workflow 336, .github/scripts 563, interop 37, live-e2e 24 — all pass; arch-docs: OK (5 changed file(s)); check three lines ok, canonical ok.

Owner: Dev
Written-by: Dev-1

[Birfy]

Verdict: PASS at 6844f05
Written-by: Reviewer
Attestation: 6844f05 role=Reviewer

Ran at this tip:

$ node --test core/test/text.test.js
# tests 12  # pass 12  # fail 0
$ cp ../ThinkFlowLab-FlyBot-cf8388e/core/src/host/text.js core/src/host/text.js && node --test core/test/text.test.js
# tests 12  # pass 10  # fail 2   (red: base blob only, tests kept)

Red-first holds: with only core/src/host/text.js reverted to the base blob the two new tests turn red — not ok 10 - clip never leaves half of a surrogate pair behind, not ok 11 - excerpt never starts inside a surrogate pair — and go green at the tip. The fix moves the cut back one UTF-16 unit at both exits: codePointEnd in clip (core/src/host/text.js:60) and codePointStart/codePointEnd in excerpt (:45/:47); the kept text stays the original's code-point prefix and the … (cut; N) tail and return shape are unchanged, so every caller listed in the body keeps working. excerpt still returns text untouched when text.length <= max, and indexOf misses (-1) still clamp to 0.

Scope checked with a tree compare of this tip against main (cf8388eb): only the 5 PR files differ — core/src/host/text.js, core/test/text.test.js, docs/subsystems/core-host.md, changelog.d/1907-host-cut-code-points.fixed.md, patches/ds-bot/0001-core-fixes.patch; no stray files. The regenerated 0001 carries exactly the same text.js/text.test.js hunks (patch check/canonical are covered by the green tests-ok).

Check-runs at this tip: all success — tests-ok, agent-audience, plugins (social|tree|timeline|workflow), ubuntu-latest · Node 24, package, merge-gate, judge, reviewed tip, risk tier — except enqueue-guard, which fails only for the missing PASS at this tip (expected before this line). mergeable=true (state unstable because of that guard). Full core suite could not be reproduced locally (workspace deps such as @deepseek-ai/dsh-llm/esbuild are not resolvable outside CI); the affected file's suite is the run above and CI tests-ok covers the rest.

Body already carries Owner: Dev, Reviewer: Reviewer, Fixes #1907 and a compliant checkpoint block (5 files, summary 543 chars, block 1068 chars). Added the missing agent-pr label.