Skip to content

Reading currency like control characters in dictionary #500

Closed
#685
Closed
Reading currency like control characters in dictionary#500
#685
Labels
bugSomething isn't working

Description

@ddehilster
ddehilster
opened on Jul 22, 2024

Currencies of the world

$ currency=1 country="United States" name="Dollar"
£ currency=1 country="United Kingdom" name="Pound Sterling"
¥ currency=1 country="Japan" name="Yen"

Activity

  1. ddehilster
    added
    bugSomething isn't working
    on Jul 22, 2024
  2. ddehilster
    added 2 commits that reference this issue on Jul 6, 2026

    fix: tokenize currency/symbol characters as punct, not control chars (#…

    55fe7c8

    fix: tokenize currency/symbol characters as punct, not control chars (#…

    75d6c4e
  3. ddehilster
    closed this as completedin #685on Jul 6, 2026

  4. ddehilster commented on Jul 6, 2026

    @ddehilster
    ddehilster
    on Jul 6, 2026
    MemberAuthor

    Fixed in #685 (merged). Currency/symbol characters ($ £ ¥ €, Unicode category Sc and other symbol categories) were falling through to the PNCTRL control-char bucket in Tok::nextTok and getting a bogus ("CTRL" <lead-byte>) attribute. Added unicu::isSymbol() and now classify them as punct (like $ always was), covering both the tokenize and dicttok passes. Note: the legacy byte-based cmltok tokenizer remains non-Unicode-aware (out of scope).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions