Repository navigation
Fix Server text output splitting multi-byte characters across service responses - #80
Merged
pcisar merged 1 commit intoOct 31, 2026
Conversation
… responses The service fills each text output response by size, not by character, so a response can end inside a multi-byte character. `_query_output()` decoded every response on its own with `read_sized_string()`, which turned both halves of such a character into U+FFFD with errors='replace', and raised UnicodeDecodeError with the default errors='strict'. Decode responses through an incremental decoder that keeps the bytes of an unfinished character until the next response. The decoder is reset with the rest of the output state in `_reset_output()`, and flushed at the end of output (and at the end of each LINE-mode line), so a character that never completes still goes to the error handler instead of being dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #79.
Problem
Server._query_output()decodes each service response on its own withread_sized_string(). The service fills text output responses by size, not by character. With a multi-byte encoding, a response can therefore end in the middle of a character. Neither half then decodes on its own:encoding_errors='replace': each half becomes U+FFFD, so a 2-byteécomes back as two replacement characters. The text is corrupted without any error.encoding_errors='strict'(the default):UnicodeDecodeErroris raised part-way through the stream, and output reading stops.Trace output is where this shows up, because it carries the SQL text of statements.
Fix
Responses are decoded through
codecs.getincrementaldecoder(self.encoding)(self.encoding_errors). The decoder keeps the bytes of an unfinished character until the next response completes it._reset_output()together with the rest of the output state. Each service call starts clean and picks up the currentencoding.final=True) at the end of output and at the end of each LINE-mode line. Bytes of a character that never completes still go to the error handler instead of disappearing, so genuinely invalid input behaves as it does today.read_bytes()isread_sized_string()without the.decode(), so nothing else changes.Evidence (live, Firebird 5.0.4 on localhost, Windows)
The probe uses the public API only:
employeewithlog_statement_finish = true.select 'ação çé ÁÉÍÓÚ ãõ' as t from rdb$database /* same text x20 */over UTF8 attachments for 3 s.mode = TO_EOF,query_timeout = 1,readline()andencoding = 'utf-8'.errors='replace', run 1errors='replace', run 2errors='strict'UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 7160: unexpected end of dataThe replacement characters always come in pairs. Each pair is one 2-byte character cut at a response boundary, for example
ação çfollowed by two U+FFFD whereéwas.Tests
test_text_output_multibyte_character_split_across_responses. It splitsçacross two responses and is parametrized over LINE/TO_EOF, with and without a timeout between the halves, and overstrict/replace.UnicodeDecodeError, or two U+FFFD in place ofç.test_text_output_incomplete_character_at_end_uses_error_handler. A character that never completes still yields U+FFFD instead of being dropped. It passes on master too; it guards the flush.output_response()test helper now also acceptsbytes.tests/test_server.pyagainst a live server: 56 passed, 1 skipped. Master gives 46 passed, 1 skipped.test_fb4.py: the server raises an integer-overflowDatabaseError.test_provider.py:OSError: [WinError 193].ruff checkgives identical findings on both changed files before and after.