LLM models leave a lot of typographic garbage and hallucinate Unicode in the middle of text. uc is a primitive filter to show or clean it.

Input: UTF-8.
Output: UTF-8.

Install via Homebrew

brew install jarpex/formulae/uc

The formula installs the latest release. To install the current main instead:

brew install --HEAD jarpex/formulae/uc

Build

make

or:

cc -std=c99 -Wall -Wextra -O2 -o uc uc.c main.c

Install

make install

The man page goes to $(PREFIX)/share/man/man1. Set PREFIX to change the destination:

make PREFIX=/usr install

Usage

usage: uc [-h] [-v] [-s] [-c] [-e encoding] [--check] [--color when]
             [--fold-diacritics] [--] [text...]

options:
  -h, --help       show help
  -v, --version    show version
  -s, --show       show problematic characters as [U+XXXX]
  -c, --clear      clean output text
  -e, --encoding   target encoding: ascii, cp1252, cp1251
      --check      exit code 1 if problems were found
      --color when colorize output: auto, always, never
      --fold-diacritics
                   fold letters with diacritics to ASCII base

-e selects allowed character set; output remains UTF-8.

Example

The input contains invisible U+00A0 characters:

printf 'foo\302\240bar\n' | uc -s

Output:

foo[U+00A0]bar

Example

cp1252 allows accented letters, so they pass through unchanged. --fold-diacritics folds them to their ASCII base instead:

printf 'caf\303\251\n' | uc -c -e cp1252 --fold-diacritics

Output:

cafe

Example

A word that mixes scripts is a homograph. The Cyrillic о in hello is repaired with the ASCII letter it mimics:

printf 'hell\320\276\n' | uc -c

Output:

hello

Test

make test

make test runs the unit tests in test.c and the CLI tests in test-cli.sh.