Skip to content

fix: make deaccent() UTF-8 aware, not Latin-1 byte-based (#488) + 3.7.5 - #689

Merged
ddehilster merged 1 commit into
masterfrom
fix/deaccent-utf8
Jul 8, 2026
Merged

ddehilster merged 1 commit into
masterfrom
fix/deaccent-utf8

Conversation

@ddehilster

Copy link
Copy Markdown
Member

Summary

Fixes #488: deaccent() mangles UTF-8 accented text — e.g. deaccent("fiancé") returned fiancA + garbage instead of fiance.

Root cause

Xml::de_accent walked the string byte-by-byte and switched on single bytes \xC0..\xFF — i.e. it assumed Latin-1. On the engine's UTF-8 text, é is two bytes (0xC3 0xA9): 0xC3 matched "Atilde" → A, and 0xA9 fell through the switch untouched. Hence fiancA<0xA9>.

Fix

Rewrite de_accent to walk UTF-8 codepoints (U8_NEXT) and map each accented codepoint to its ASCII base:

  • The old byte table's values (0xC0..0xFF) are exactly the Latin-1 Supplement codepoints U+00C0..U+00FF, so the mapping carries over unchanged.
  • The 0x8A..0x9F Windows-1252 letters map to their real Unicode codepoints (Scaron U+0160, OElig U+0152, …).
  • ASCII bytes are copied unchanged → de-accent matching on ASCII is byte-identical to before (no regression); unmapped multi-byte codepoints are preserved verbatim.
  • Replaces the byte-based de_accent_ch (removed; it was used only by these loops).

Verified

Input Output
fiancé fiance ✅
café / naïve cafe / naive ✅
señor / Zürich / garçon senor / Zurich / garcon ✅
Œuvre Euvre ✅
hello (ASCII) hello (unchanged) ✅

emailaddress analyzer still loads/extracts — no regression. de_accent is also used by rule-matching (Arun.cpp); since ASCII output is unchanged, existing matching is unaffected and UTF-8 matching now de-accents correctly.

⚠️ Merge order: bumps → 3.7.5, assuming #688 (→3.7.4) merges first. If merged before #688, keep 3.7.5 when resolving the trivial main.cpp version-line conflict.

Fixes #488.

🤖 Generated with Claude Code

Xml::de_accent walked the string byte-by-byte and switched on single bytes
(\xC0..\xFF), i.e. it assumed Latin-1. On the engine's UTF-8 text a letter like
e-acute is two bytes (0xC3 0xA9): 0xC3 matched "Atilde" -> "A" and 0xA9 fell
through untouched, so deaccent("fiancé") produced "fiancA<garbage>".

Rewrite de_accent to walk UTF-8 codepoints (U8_NEXT) and map the accented
codepoint to its ASCII base. The old byte table's values (0xC0..0xFF) are
exactly the Latin-1 Supplement codepoints U+00C0..U+00FF, so the mapping carries
over unchanged; the 0x8A..0x9F Windows-1252 letters map to their real Unicode
codepoints (Scaron U+0160, OElig U+0152, ...). ASCII bytes are copied unchanged
(so de-accent matching on ASCII is byte-identical to before); unmapped multi-byte
codepoints are preserved verbatim. Replaces the byte-based de_accent_ch.

Verified: fiancé->fiance, café->cafe, naïve->naive, señor->senor, Zürich->Zurich,
garçon->garcon, Œuvre->Euvre, hello->hello. emailaddress analyzer unchanged.

Bumps NLP_ENGINE_VERSION to 3.7.5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ddehilster
ddehilster force-pushed the fix/deaccent-utf8 branch from d0d37a7 to fac93a5 Compare July 8, 2026 00:38
@ddehilster
ddehilster merged commit 7b2768d into master Jul 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

daccent function not working correctly

1 participant