Skip to content

Fix Server text output splitting multi-byte characters across service responses - #80

Merged
pcisar merged 1 commit into
FirebirdSQL:masterfrom
fdcastel:fix-output-multibyte-chunk-boundary
Oct 31, 2026
Merged

pcisar merged 1 commit into
FirebirdSQL:masterfrom
fdcastel:fix-output-multibyte-chunk-boundary

Conversation

@fdcastel

Copy link
Copy Markdown
Member

Follow-up to #79.

Problem

Server._query_output() decodes each service response on its own with read_sized_string(). The service fills text output responses by size, not by character. With a multi-byte encoding, a response can therefore end in the middle of a character. Neither half then decodes on its own:

  • encoding_errors='replace': each half becomes U+FFFD, so a 2-byte é comes back as two replacement characters. The text is corrupted without any error.
  • encoding_errors='strict' (the default): UnicodeDecodeError is raised part-way through the stream, and output reading stops.

Trace output is where this shows up, because it carries the SQL text of statements.

Fix

Responses are decoded through codecs.getincrementaldecoder(self.encoding)(self.encoding_errors). The decoder keeps the bytes of an unfinished character until the next response completes it.

  • It is created on the first response and reset in _reset_output() together with the rest of the output state. Each service call starts clean and picks up the current encoding.
  • It is flushed (final=True) at the end of output and at the end of each LINE-mode line. Bytes of a character that never completes still go to the error handler instead of disappearing, so genuinely invalid input behaves as it does today.

read_bytes() is read_sized_string() without the .decode(), so nothing else changes.

Evidence (live, Firebird 5.0.4 on localhost, Windows)

The probe uses the public API only:

  • A trace session on employee with log_statement_finish = true.
  • Four threads run select 'ação çé ÁÉÍÓÚ ãõ' as t from rdb$database /* same text x20 */ over UTF8 attachments for 3 s.
  • The output is read with mode = TO_EOF, query_timeout = 1, readline() and encoding = 'utf-8'.
Run 2.1.1 this PR
errors='replace', run 1 108 U+FFFD in 54 lines (112,661 lines, 10.6 MB) 0 (108,283 lines, 10.2 MB)
errors='replace', run 2 120 U+FFFD in 60 lines (113,673 lines) 0 (116,005 lines)
errors='strict' UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 7160: unexpected end of data 0 (90,584 lines)

The replacement characters always come in pairs. Each pair is one 2-byte character cut at a response boundary, for example ação ç followed by two U+FFFD where é was.

Tests

  • New test_text_output_multibyte_character_split_across_responses. It splits ç across two responses and is parametrized over LINE/TO_EOF, with and without a timeout between the halves, and over strict/replace.
    • On master, all 8 cases fail: UnicodeDecodeError, or two U+FFFD in place of ç.
    • With the fix, all 8 pass.
  • New test_text_output_incomplete_character_at_end_uses_error_handler. A character that never completes still yields U+FFFD instead of being dropped. It passes on master too; it guards the flush.
  • The output_response() test helper now also accepts bytes.
  • tests/test_server.py against a live server: 56 passed, 1 skipped. Master gives 46 passed, 1 skipped.
  • Full suite: 173 passed, 16 failed, 2 skipped. Master gives 163 passed with the same 16 failures (identical test IDs). On my machine those are all environmental:
    • 8 in test_fb4.py: the server raises an integer-overflow DatabaseError.
    • 8 in test_provider.py: OSError: [WinError 193].
  • ruff check gives identical findings on both changed files before and after.

… responses

The service fills each text output response by size, not by character, so a
response can end inside a multi-byte character. `_query_output()` decoded every
response on its own with `read_sized_string()`, which turned both halves of
such a character into U+FFFD with errors='replace', and raised
UnicodeDecodeError with the default errors='strict'.

Decode responses through an incremental decoder that keeps the bytes of an
unfinished character until the next response. The decoder is reset with the
rest of the output state in `_reset_output()`, and flushed at the end of output
(and at the end of each LINE-mode line), so a character that never completes
still goes to the error handler instead of being dropped.
@pcisar
pcisar merged commit a676c89 into FirebirdSQL:master Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants