Skip to content

programs: strip trailing '\r' from --filelist entries on Windows - #4766

Merged
Cyan4973 merged 2 commits into
meta:devfrom
jaypatrickhoward:filelist-crlf
Nov 15, 2026
Merged

Cyan4973 merged 2 commits into
meta:devfrom
jaypatrickhoward:filelist-crlf

Conversation

@jaypatrickhoward

@jaypatrickhoward jaypatrickhoward commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

On Windows, --filelist no longer accepts file lists that use CRLF line endings. Every path retains a trailing \r, so each entry fails to resolve:

> printf 'a.txt\r\nb.txt\r\n' > list.txt
> zstd --filelist=list.txt
zstd: can't stat a.txt : No such file or directory -- ignored

The \r is invisible in that message, which makes the failure awkward to diagnose.

Cause

The list used to be opened in text mode (fopen(..., "r")), where the Windows CRT translates CRLF to LF before the parser sees it, so stripping the trailing '\n' in readLineFromFile() was sufficient. There was never an explicit '\r' strip — the behaviour depended entirely on that translation.

#4349 (165e52ce, "first implementation supporting Process Substitution") reads the whole list into a buffer opened in binary mode and splits it on '\n' alone. Binary mode performs no translation, so the CR now survives into every filename. Because the previous behaviour was implicit, nothing flagged its loss.

Binary mode is the right choice for the new implementation, so the fix is to strip the CR explicitly.

Fix

Drop a trailing '\r' in UTIL_createLinePointers(), where each line's extent is already known — but only on Windows.

The guard matters, because the two interesting inputs are byte-identical. A CRLF-terminated line naming a.txt, and an LF-terminated line naming a file actually called a.txt\r, are both a.txt \r \n. No parser can distinguish them, so stripping unconditionally would trade one behaviour for the other. Guarding on _WIN32 picks the interpretation that is correct on each platform:

  • On Windows, CRLF is the native separator, and a filename ending in '\r' was never usable by zstd in the first place — see the note below the table. So stripping loses nothing.
  • Elsewhere, '\r' is a legal byte in a filename, so it is left untouched.

This reproduces the pre-regression behaviour exactly, on both platforms, without introducing a new one.

Measured behaviour

One probe script, run on both platforms; zstd.exe built with MSVC (x64, Release).

before #4349 current dev this patch
CRLF list, Windows works fails works
CRLF list, POSIX fails fails fails
filename ending in '\r', Windows fails fails fails
filename ending in '\r', POSIX works works works

Every cell matches the pre-#4349 column.

The third row deserves a note, since it is the one case this patch gives up. Such a filename can exist on NTFS — the probe's emulation layer did create one — but no zstd build can open it, including before #4349: Win32 reserves the characters 0x01-0x1F in path names, so a native build never had access to it. Stripping the CR on Windows therefore removes no functionality that was previously available; it only stops a CRLF list from being misread.

A CRLF list authored on Windows still fails when consumed on a POSIX host, as it always has. Making that work would be a new feature rather than a regression fix, and it cannot be done without giving up filenames ending in '\r', so it is left alone here.

Scope

Not yet released. 165e52ce is not an ancestor of v1.5.7, v1.5.6 or v1.5.5, and no tag contains it; it is an ancestor of dev only. This is a dev regression that would first appear in the next release.

Testing

A playTests.sh case is added for the CRLF list, annotated with #4349 so a future reader knows why it exists. It runs only where the strip is compiled in — MINGW*/MSYS*, matching the split tests/Makefile already makes (it groups CYGWIN_NT% with Linux and Darwin, and sets HOST_OS=MSYS for MINGW%/MSYS%).

  • With the fix reverted, an MSVC build fails this case: playTests.sh aborts at exactly that test, reporting can't stat for both entries.
  • With the fix, it passes in visual-runtime-tests on both x64 and Win32 — the log shows the case executing and 2 files compressed.
  • Full playTests.sh passes on Windows and on POSIX.
  • POSIX behaviour is byte-identical to dev: 300 randomised list shapes, 147 of which contain CR, compared on exit status, files produced and stderr, with no differences. Expected, since the strip is compiled out there.
  • The table above was produced by building 165e52ce~1, dev, and this patch, then running one probe script on both platforms.

@meta-cla meta-cla Bot added the CLA Signed label Sep 7, 2026
@jaypatrickhoward
jaypatrickhoward marked this pull request as draft September 7, 2026 05:06
@jaypatrickhoward jaypatrickhoward changed the title programs: strip trailing '\r' from --filelist entries programs: strip trailing '\r' from --filelist entries on Windows Sep 7, 2026
@jaypatrickhoward
jaypatrickhoward force-pushed the filelist-crlf branch 8 times, most recently from 078603b to e74a51a Compare September 7, 2026 13:34
@jaypatrickhoward
jaypatrickhoward marked this pull request as ready for review September 7, 2026 13:57
`--filelist` stopped accepting Windows CRLF line endings in 165e52c
("first implementation supporting Process Substitution", meta#4349). That
rewrite reads the list into a buffer opened in binary mode and splits it
on '\n' alone. Previously the list was opened in text mode, where the
Windows CRT translated CRLF to LF before the parser saw it, so stripping
'\n' was sufficient. In binary mode the CR survives into every path:

    zstd: can't stat a.txt : No such file or directory -- ignored

The '\r' is invisible in that message, which makes it awkward to
diagnose.

Strip a trailing '\r' when the line pointers are built, but only on
Windows, where CRLF is the native line separator and a '\r' cannot be
used in a filename. Elsewhere '\r' is a legal filename byte and is left
alone, so behaviour on those platforms is unchanged. This restores the
pre-regression behaviour exactly, on every platform, without adding a
new one.

Adds a playTests case, guarded to Windows, that fails without the fix.
@Cyan4973 Cyan4973 self-assigned this Nov 15, 2026
The CRLF test only ran when the shell was MINGW* or MSYS*. Native Windows
builds are now detected once, as Windows minus Cygwin (the only Windows
environment that doesn't define _WIN32), so new MSYS2 environment names
don't silently skip the test.

The test also checks the other half of the behavior: elsewhere, a file
name ending with CR is kept, so the CR strip stays limited to _WIN32.
@Cyan4973
Cyan4973 merged commit 7f02a28 into meta:dev Nov 15, 2026
109 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants