Repository navigation
fix: make deaccent() UTF-8 aware, not Latin-1 byte-based (#488) + 3.7.5 - #689
Merged
Merged
Conversation
Xml::de_accent walked the string byte-by-byte and switched on single bytes
(\xC0..\xFF), i.e. it assumed Latin-1. On the engine's UTF-8 text a letter like
e-acute is two bytes (0xC3 0xA9): 0xC3 matched "Atilde" -> "A" and 0xA9 fell
through untouched, so deaccent("fiancé") produced "fiancA<garbage>".
Rewrite de_accent to walk UTF-8 codepoints (U8_NEXT) and map the accented
codepoint to its ASCII base. The old byte table's values (0xC0..0xFF) are
exactly the Latin-1 Supplement codepoints U+00C0..U+00FF, so the mapping carries
over unchanged; the 0x8A..0x9F Windows-1252 letters map to their real Unicode
codepoints (Scaron U+0160, OElig U+0152, ...). ASCII bytes are copied unchanged
(so de-accent matching on ASCII is byte-identical to before); unmapped multi-byte
codepoints are preserved verbatim. Replaces the byte-based de_accent_ch.
Verified: fiancé->fiance, café->cafe, naïve->naive, señor->senor, Zürich->Zurich,
garçon->garcon, Œuvre->Euvre, hello->hello. emailaddress analyzer unchanged.
Bumps NLP_ENGINE_VERSION to 3.7.5.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ddehilster
force-pushed
the
fix/deaccent-utf8
branch
from
July 8, 2026 00:38
d0d37a7 to
fac93a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #488:
deaccent()mangles UTF-8 accented text — e.g.deaccent("fiancé")returnedfiancA+ garbage instead offiance.Root cause
Xml::de_accentwalked the string byte-by-byte and switched on single bytes\xC0..\xFF— i.e. it assumed Latin-1. On the engine's UTF-8 text,éis two bytes (0xC3 0xA9):0xC3matched "Atilde" →A, and0xA9fell through the switch untouched. HencefiancA<0xA9>.Fix
Rewrite
de_accentto walk UTF-8 codepoints (U8_NEXT) and map each accented codepoint to its ASCII base:0xC0..0xFF) are exactly the Latin-1 Supplement codepointsU+00C0..U+00FF, so the mapping carries over unchanged.0x8A..0x9FWindows-1252 letters map to their real Unicode codepoints (ScaronU+0160, OEligU+0152, …).de_accent_ch(removed; it was used only by these loops).Verified
fiancéfiance✅café/naïvecafe/naive✅señor/Zürich/garçonsenor/Zurich/garcon✅ŒuvreEuvre✅hello(ASCII)hello(unchanged) ✅emailaddressanalyzer still loads/extracts — no regression.de_accentis also used by rule-matching (Arun.cpp); since ASCII output is unchanged, existing matching is unaffected and UTF-8 matching now de-accents correctly.main.cppversion-line conflict.Fixes #488.
🤖 Generated with Claude Code