Conversation
6 tasks
Rebased onto main: upstream moved the tree under python/; the fix applies unchanged to python/src/edge0/server/app.py.
ahrazzle
force-pushed
the
fix/sse-incremental-decode
branch
from
October 3, 2026 18:26
72586c0 to
690724a
Compare
Resolve the conflict in _chat_stream: keep main's tool-call buffering path (buffer the full response, parse once, emit a single delta with tool_calls and finish_reason=tool_calls) and layer the incremental SSE decode on top of the non-buffering path -- on_token appends to seen_ids, decodes the cumulative id list, and emits only the new suffix via _incremental_suffix, with a final flush so concatenated deltas equal the non-streaming text exactly. ToolCallTok in the tests now decodes prefix-stably (one fixed piece per id, like a real byte-level BPE tokenizer) so the no-tools regression guard still observes one immediate delta per token under cumulative decoding. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Streamed SSE responses now decode the full id list cumulatively and emit only the newly completed suffix, holding back partial multi-byte characters until they resolve.
Root cause
The streaming path decoded each token id in isolation (python/src/edge0/server/app.py:82). A byte-level BPE token carrying part of a multi-byte character streamed as U+FFFD while the non-streaming path returned correct text.
Changes
Rebase needed
The multi-platform layout refactor (cad29a1) moved the touched files to python/src/edge0/server/app.py and python/tests/test_server.py. This branch predates it and no longer applies (mergeable: conflicting). Rebase onto main first. The diff carries over unchanged: a local rebase onto feafe31 applied with rename detection and zero content conflicts.
Validation
Live edge0-8b server run (stdlib transport), prompt asking the model to echo 🌊🏖️🦀🍣:
The non-streaming path decodes the whole id list and was already correct.
Test: python/tests/test_server.py::test_chat_stream_incremental_decode_of_split_characters
Limits
Common CJK text in the shipped vocabulary is not affected. Corruption needs characters that split across byte tokens (emoji, rare CJK, some Korean).
O(n²) cumulative re-decode per request. This is the minimal fix. Throughput was not measured.