Repository navigation
[cli] Support non-ASCII file names on Windows (fixes #3882) - #4973
Draft
kevinjzhang wants to merge 4 commits into
Draft
kevinjzhang wants to merge 4 commits into
kevinjzhang wants to merge 4 commits into
Conversation
Handle file names as UTF-8 internally and convert to UTF-16 at the OS boundary (command line, file access, directory listing, console output), following the approach used by xxHash (Cyan4973/xxHash#293). Fixes meta#3882
added 3 commits
November 16, 2026 14:10
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3882
Why zstd fails on some Windows file names today
Windows has two parallel sets of file/console APIs. The "narrow" (
char*) ones that zstd uses (fopen,_stat64,_open,remove,_chmod,utime,_mkdir,FindFirstFileA,GetFileAttributesA,CreateFileA,main(argv)) interpret strings in the active ANSI code page (ACP, e.g. Windows-1252 on a western install). Only the "wide" (wchar_t*, UTF-16) ones can represent every file name. Three consequences:main()'sargvis already converted UTF-16 → ACP by the C runtime. Any character not in the ACP is replaced by?(or a best-fit) before zstd sees it, so the file cannot be found/opened (xxHash hit exactly this:Could not open '????.txt': Invalid argument).fopen/stat, it is mapped through the ACP.FindFirstFileA/FindNextFileA(used by-rand directory inputs) return names converted UTF-16 → ACP, so files whose names contain non-ACP characters come back as?-names that cannot be opened and are skipped or error out. This is the "zstd silently misses files in a folder" behaviour reported in the thread.fprintf, which the console interprets in the OEM code page, not the ACP. A name that round-trips fine through the ACP is still displayed wrongly. This is the exact symptom in the report:öis0xF6in Windows-1252 but0xF6is÷in OEM CP437, sovögel.txtprints asv÷gel.txt(whilevögel.txt.zstwas created correctly).With MinGW there is a fourth wrinkle: its C runtime expands wildcards (
zstd *.*) while building the ACPargv, so the expanded names are lossy as in (1).(Items 1 and 3 are directly supported by the issue and the xxHash thread; item 2 follows from the documented behaviour of the
*AAPIs. I have not reproduced them on a Windows machine myself.)Approach
Same approach as xxHash (Cyan4973/xxHash#293, fixed by #304 and #323): treat every file name as UTF-8 internally and convert to UTF-16 only at the OS boundary.
wmain()on MSVC;main()+GetCommandLineW()/CommandLineToArgvW()elsewhere (MinGW without-municode).argvis converted to UTF-8 before the existing CLI runs (zstdcli.c).UTIL_fopen,UTIL_open,UTIL_removewrappers over_wfopen/_wopen/_wremove;UTIL_stat/chmod/utime/mkdiruse their_w*variants;GetFileAttributesW,FindFirstFileW/FindNextFileW,CreateFileWfor link detection, directory recursion and dictionary mmap.UTIL_fprintf. When the stream is a console it formats as UTF-8, converts to UTF-16 and prints viafwprintfwith the stream temporarily in_O_U8TEXT(thefprintf_utf8technique from xxHash#323).DISPLAY*macros use it.On non-Windows platforms the new helpers are macros for
fopen/remove/fprintf: no behavior change.Also fixes a small leak in the Windows
UTIL_prepareFileList(pathwas not freed when skipping./..).Deliberate difference from xxHash
xxHash switches stdout/stderr to
_O_U8TEXTfor the whole process. That is not possible here: zstd writes binary data to stdout (zstd -c,zstdcat), and narrow writes on a_O_U8TEXTstream assert.UTIL_fprintftherefore only converts when the stream is an actual console (GetConsoleMode), and passes raw UTF-8 through unchanged when output is a file or pipe. This also avoids the code-page-dependent output reported against xxHash.Tests
tests/playTests.shgains a "non-ASCII file names" section. The names are deliberately outside the ACP of a western Windows install (Japaneseユニコード, Cyrillicпривет) plus Latin-1vögel.txtfrom the report, because a Latin-1 name alone would round-trip through the ANSI APIs and not exercise the fix. It covers every boundary changed:-t/ decompress round trip, name reported as-is in-voutput (console/fprintfpath, via a pipe)-ooutput name,--rm(remove), andzstd -c | zstd -dcstays binary-clean (guards the stdout concern above)-rcompress and-d -r --rm(theFindFirstFileWtraversal),--filelist,--output-dir-flat-Dand--patch-fromResults:
zstd,fullbench,fuzzer,paramgrillbuild cleanly; the newplayTests.shsection passes.mingw-short-test(gcc MINGW32 / MINGW64):make checkrunsplayTests.shnatively againstzstd.exe, including the new non-ASCII section (confirmed in the job logs,isWindows=true). The clang MINGW64 variant (-Werror -Wconversion) also builds.UTIL_fprintf(_O_U8TEXTviafwprintf). CI output is piped, and in that caseUTIL_fprintfdeliberately writes raw UTF-8, so the console branch is never exercised. Manual verification in a real Windows console (cmd / PowerShell / Windows Terminal, withchcp 437andchcp 65001) would be appreciated.Known limitations / open questions
GetCommandLineW()bypasses the MinGW CRT's*.*expansion, so MinGW builds will no longer glob (MSVC builds never did). xxHash accepted the same trade-off. To keep it we could use-municode+wmain(the CRT globswmainargs too) or__wgetmainargs(..., _dowildcard, ...). Feedback welcome.--filelistcontent is read as bytes and used as file names, so it is now interpreted as UTF-8 instead of the ANSI code page. A list file saved in the ANSI code page that contained non-ASCII names, and worked before, will no longer match those files. ASCII-only lists are unaffected. A UTF-8 BOM is still not stripped (unchanged).öwas0xF6, now0xC3 0xB6). That is arguably more correct, but scripts that parse zstd's output on Windows may notice. Console output is unaffected apart from now being displayed correctly.getenv-style Win32 calls not tied to file names are unchanged;benchfn.c's DEBUG-onlyDISPLAYis unchanged.