Skip to content

[cli] Support non-ASCII file names on Windows (fixes #3882) - #4973

Draft
kevinjzhang wants to merge 4 commits into
meta:devfrom
kevinjzhang:fix-windows-utf8-filenames
Draft

kevinjzhang wants to merge 4 commits into
meta:devfrom
kevinjzhang:fix-windows-utf8-filenames

Conversation

@kevinjzhang

@kevinjzhang kevinjzhang commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3882

Why zstd fails on some Windows file names today

Windows has two parallel sets of file/console APIs. The "narrow" (char*) ones that zstd uses (fopen, _stat64, _open, remove, _chmod, utime, _mkdir, FindFirstFileA, GetFileAttributesA, CreateFileA, main(argv)) interpret strings in the active ANSI code page (ACP, e.g. Windows-1252 on a western install). Only the "wide" (wchar_t*, UTF-16) ones can represent every file name. Three consequences:

  1. Lossy command line. main()'s argv is already converted UTF-16 → ACP by the C runtime. Any character not in the ACP is replaced by ? (or a best-fit) before zstd sees it, so the file cannot be found/opened (xxHash hit exactly this: Could not open '????.txt': Invalid argument).
  2. Lossy file access and directory traversal. Even if a name reaches fopen/stat, it is mapped through the ACP. FindFirstFileA/FindNextFileA (used by -r and directory inputs) return names converted UTF-16 → ACP, so files whose names contain non-ACP characters come back as ?-names that cannot be opened and are skipped or error out. This is the "zstd silently misses files in a folder" behaviour reported in the thread.
  3. Wrong console output. Names are written to the console with narrow fprintf, which the console interprets in the OEM code page, not the ACP. A name that round-trips fine through the ACP is still displayed wrongly. This is the exact symptom in the report: ö is 0xF6 in Windows-1252 but 0xF6 is ÷ in OEM CP437, so vögel.txt prints as v÷gel.txt (while vögel.txt.zst was created correctly).

With MinGW there is a fourth wrinkle: its C runtime expands wildcards (zstd *.*) while building the ACP argv, so the expanded names are lossy as in (1).

(Items 1 and 3 are directly supported by the issue and the xxHash thread; item 2 follows from the documented behaviour of the *A APIs. I have not reproduced them on a Windows machine myself.)

Approach

Same approach as xxHash (Cyan4973/xxHash#293, fixed by #304 and #323): treat every file name as UTF-8 internally and convert to UTF-16 only at the OS boundary.

Boundary Change
Command line wmain() on MSVC; main() + GetCommandLineW()/CommandLineToArgvW() elsewhere (MinGW without -municode). argv is converted to UTF-8 before the existing CLI runs (zstdcli.c).
File access New UTIL_fopen, UTIL_open, UTIL_remove wrappers over _wfopen/_wopen/_wremove; UTIL_stat/chmod/utime/mkdir use their _w* variants; GetFileAttributesW, FindFirstFileW/FindNextFileW, CreateFileW for link detection, directory recursion and dictionary mmap.
Console output New UTIL_fprintf. When the stream is a console it formats as UTF-8, converts to UTF-16 and prints via fwprintf with the stream temporarily in _O_U8TEXT (the fprintf_utf8 technique from xxHash#323). DISPLAY* macros use it.

On non-Windows platforms the new helpers are macros for fopen/remove/fprintf: no behavior change.

Also fixes a small leak in the Windows UTIL_prepareFileList (path was not freed when skipping ./..).

Deliberate difference from xxHash

xxHash switches stdout/stderr to _O_U8TEXT for the whole process. That is not possible here: zstd writes binary data to stdout (zstd -c, zstdcat), and narrow writes on a _O_U8TEXT stream assert. UTIL_fprintf therefore only converts when the stream is an actual console (GetConsoleMode), and passes raw UTF-8 through unchanged when output is a file or pipe. This also avoids the code-page-dependent output reported against xxHash.

Tests

tests/playTests.sh gains a "non-ASCII file names" section. The names are deliberately outside the ACP of a western Windows install (Japanese ユニコード, Cyrillic привет) plus Latin-1 vögel.txt from the report, because a Latin-1 name alone would round-trip through the ANSI APIs and not exercise the fix. It covers every boundary changed:

  • compress / -t / decompress round trip, name reported as-is in -v output (console/fprintf path, via a pipe)
  • explicit -o output name, --rm (remove), and zstd -c | zstd -dc stays binary-clean (guards the stdout concern above)
  • non-ASCII directory with -r compress and -d -r --rm (the FindFirstFileW traversal), --filelist, --output-dir-flat
  • non-ASCII dictionary path with -D and --patch-from

Results:

  • macOS (clang): zstd, fullbench, fuzzer, paramgrill build cleanly; the new playTests.sh section passes.
  • Windows CI: all 109 checks pass on the latest commit, including the jobs that matter most here:
    • mingw-short-test (gcc MINGW32 / MINGW64): make check runs playTests.sh natively against zstd.exe, including the new non-ASCII section (confirmed in the job logs, isWindows=true). The clang MINGW64 variant (-Werror -Wconversion) also builds.
    • MSVC x64/Win32/ARM64, Clang-CL, MSBuild (VS 2022/2025), MinGW CMake, Cygwin, and the mingw cross-compile jobs.
  • Not covered by CI: the console path of UTIL_fprintf (_O_U8TEXT via fwprintf). CI output is piped, and in that case UTIL_fprintf deliberately writes raw UTF-8, so the console branch is never exercised. Manual verification in a real Windows console (cmd / PowerShell / Windows Terminal, with chcp 437 and chcp 65001) would be appreciated.
  • I have not shown that the new tests fail without the fix on Windows (a negative control); I only ran them against the fixed build.

Known limitations / open questions

  • Wildcard expansion with MinGW. Reading GetCommandLineW() bypasses the MinGW CRT's *.* expansion, so MinGW builds will no longer glob (MSVC builds never did). xxHash accepted the same trade-off. To keep it we could use -municode + wmain (the CRT globs wmain args too) or __wgetmainargs(..., _dowildcard, ...). Feedback welcome.
  • File lists must be UTF-8. --filelist content is read as bytes and used as file names, so it is now interpreted as UTF-8 instead of the ANSI code page. A list file saved in the ANSI code page that contained non-ASCII names, and worked before, will no longer match those files. ASCII-only lists are unaffected. A UTF-8 BOM is still not stripped (unchanged).
  • Redirected/piped output is now UTF-8. When stderr/stdout is a file or pipe, file names are written as UTF-8 bytes instead of ANSI-code-page bytes (e.g. ö was 0xF6, now 0xC3 0xB6). That is arguably more correct, but scripts that parse zstd's output on Windows may notice. Console output is unaffected apart from now being displayed correctly.
  • getenv-style Win32 calls not tied to file names are unchanged; benchfn.c's DEBUG-only DISPLAY is unchanged.

Handle file names as UTF-8 internally and convert to UTF-16 at the OS
boundary (command line, file access, directory listing, console output),
following the approach used by xxHash (Cyan4973/xxHash#293).

Fixes meta#3882
@meta-cla meta-cla Bot added the CLA Signed label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zstd fails to process some filenames on Windows [we need a hero]

1 participant