TarReader fails to read tar file with hardlinks; throws System.IO.EndOfStreamException

[am11]

Try to read links-strip.tar from https://github.com/npm/node-tar/tree/f12992932f171ea248b27fad95e7d489a56d31ed/test/fixtures/tars:

Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.Stream.ReadAtLeastCore(Span`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream)
   at System.IO.Stream.ReadExactly(Span`1 buffer)
   at System.Formats.Tar.TarHeader.TryGetNextHeader(Stream archiveStream, Boolean copyData, TarEntryFormat initialFormat)
   at System.Formats.Tar.TarReader.TryGetNextEntryHeader(Boolean copyData)
   at System.Formats.Tar.TarReader.GetNextEntry(Boolean copyData)
   at Program.<<Main>$>g__ReadTar|0_0(String path, Boolean useGz) in /home/am11/projects/consoletar/Program.cs:line 26
   at Program.<Main>$(String[] args) in /home/am11/projects/consoletar/Program.cs:line 15

Repro:

# bash on linux-x64

# daily build
$ dotnet7 --version
7.0.100-rc.2.22419.21

$ dotnet7 new console -n consoletar; cd $_
$ cat > Program.cs <<EOF
using System;
using System.Formats.Tar;
using System.IO;
using System.IO.Compression;

foreach (string file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tgz", SearchOption.AllDirectories))
{
    Console.WriteLine("{0} ->", file);
    ReadTar(file, true);
}

foreach (string file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tar", SearchOption.AllDirectories))
{
    Console.WriteLine("{0} ->", file);
    ReadTar(file);
}

Console.WriteLine("Done");

void ReadTar(string path, bool useGz = false)
{
    using FileStream sourceStream = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read);
    using var reader = new TarReader(useGz ? new GZipStream(sourceStream, CompressionMode.Decompress) : sourceStream);

    TarEntry? entry = null;
    while ((entry = reader.GetNextEntry()) is not null)
    {
        Console.WriteLine("\t{0} ({1} bytes)", entry.Name, entry.Length);
    }
}
EOF

$ git clone https://github.com/npm/node-tar

$ dotnet7 run
...
Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
...

# test with GNU tar(1)
$ mkdir temp
$ tar xzf node-tar/test/fixtures/tars/links-strip.tar -C temp
# succeeds

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Try to read links-strip.tar from https://github.com/npm/node-tar/tree/f12992932f171ea248b27fad95e7d489a56d31ed/test/fixtures/tars:

Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.Stream.ReadAtLeastCore(Span`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream)
   at System.IO.Stream.ReadExactly(Span`1 buffer)
   at System.Formats.Tar.TarHeader.TryGetNextHeader(Stream archiveStream, Boolean copyData, TarEntryFormat initialFormat)
   at System.Formats.Tar.TarReader.TryGetNextEntryHeader(Boolean copyData)
   at System.Formats.Tar.TarReader.GetNextEntry(Boolean copyData)
   at Program.<<Main>$>g__ReadTar|0_0(String path, Boolean useGz) in /home/am11/projects/consoletar/Program.cs:line 26
   at Program.<Main>$(String[] args) in /home/am11/projects/consoletar/Program.cs:line 15

Repro:

# bash on linux-x64

# daily build
$ dotnet7 --version
7.0.100-rc.2.22419.21

$ dotnet7 new console -n consoletar; cd $_
$ cat > Program.cs <<EOF
using System;
using System.IO;
using System.IO.Compression;
using System.Formats.Tar;

foreach (string file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tgz", SearchOption.AllDirectories))
{
    Console.WriteLine("{0} ->", file);
    ReadTar(file, true);
}

foreach (string file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tar", SearchOption.AllDirectories))
{
    Console.WriteLine("{0} ->", file);
    ReadTar(file);
}

Console.WriteLine("Done");

void ReadTar(string path, bool useGz = false)
{
    using FileStream sourceStream = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read);
    using var reader = new TarReader(useGz ? new GZipStream(sourceStream, CompressionMode.Decompress) : sourceStream);

    TarEntry? entry = null;
    while ((entry = reader.GetNextEntry()) is not null)
    {
        Console.WriteLine("\t{0} ({1} bytes)", entry.Name, entry.Length);
    }
}
EOF

$ git clone https://github.com/npm/node-tar

$ dotnet7 run
...
Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
...

# test with GNU tar(1)
$ mkdir temp
$ tar xzf node-tar/test/fixtures/tars/links-strip.tar -C temp
# succeeds

Author:	am11
Assignees:	-
Labels:	area-System.IO.Compression
Milestone:	-

[am11]

For the lack of area-System.Formats.Tar label, assigned area-System.IO.Compression.

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Try to read links-strip.tar from https://github.com/npm/node-tar/tree/f12992932f171ea248b27fad95e7d489a56d31ed/test/fixtures/tars:

Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.Stream.ReadAtLeastCore(Span`1 buffer, Int32 minimumBytes, Boolean throwOnEndOfStream)
   at System.IO.Stream.ReadExactly(Span`1 buffer)
   at System.Formats.Tar.TarHeader.TryGetNextHeader(Stream archiveStream, Boolean copyData, TarEntryFormat initialFormat)
   at System.Formats.Tar.TarReader.TryGetNextEntryHeader(Boolean copyData)
   at System.Formats.Tar.TarReader.GetNextEntry(Boolean copyData)
   at Program.<<Main>$>g__ReadTar|0_0(String path, Boolean useGz) in /home/am11/projects/consoletar/Program.cs:line 26
   at Program.<Main>$(String[] args) in /home/am11/projects/consoletar/Program.cs:line 15

Repro:

# bash on linux-x64

# daily build
$ dotnet7 --version
7.0.100-rc.2.22419.21

$ dotnet7 new console -n consoletar; cd $_
$ cat > Program.cs <<EOF
using System;
using System.Formats.Tar;
using System.IO;
using System.IO.Compression;

foreach (string file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tgz", SearchOption.AllDirectories))
{
    Console.WriteLine("{0} ->", file);
    ReadTar(file, true);
}

foreach (string file in Directory.EnumerateFiles(Environment.CurrentDirectory, "*.tar", SearchOption.AllDirectories))
{
    Console.WriteLine("{0} ->", file);
    ReadTar(file);
}

Console.WriteLine("Done");

void ReadTar(string path, bool useGz = false)
{
    using FileStream sourceStream = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read);
    using var reader = new TarReader(useGz ? new GZipStream(sourceStream, CompressionMode.Decompress) : sourceStream);

    TarEntry? entry = null;
    while ((entry = reader.GetNextEntry()) is not null)
    {
        Console.WriteLine("\t{0} ({1} bytes)", entry.Name, entry.Length);
    }
}
EOF

$ git clone https://github.com/npm/node-tar

$ dotnet7 run
...
Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
...

# test with GNU tar(1)
$ mkdir temp
$ tar xzf node-tar/test/fixtures/tars/links-strip.tar -C temp
# succeeds

Author:	am11
Assignees:	-
Labels:	area-System.IO, untriaged
Milestone:	-

[danmoseley]

We settled on System.IO label

[danmoseley]

Any interest in debugging a bit @am11 ?

@carlossanlop did we try to assemble a corpus of tar files to breadth test unarchiving? There must be some way to gather a variety of them. Perhaps other tar implementations have a test bed we can reuse. Then we just put the code in a VM and loop over decompressing each in turn to see what blows up.

[am11]

Any interest in debugging a bit @am11 ?

A pragmatic fix is:

--- a/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarReader.cs
+++ b/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarReader.cs
@@ -138,6 +138,11 @@ public async ValueTask DisposeAsync()
                     TarEntryFormat.V7 or TarEntryFormat.Unknown or _ => new V7TarEntry(header, this),
                 };
 
+                if (_archiveStream.CanSeek && _archiveStream.Length == _archiveStream.Position)
+                {
+                    _reachedEndMarkers = true;
+                }
+
                 _previouslyReadEntry = entry;
                 PreserveDataStreamForDisposalIfNeeded(entry);
                 return entry;

but I will defer to @carlossanlop since he is looking into other similar issues based on the discussion in #74281.

After fixing the EndOfStreamException, found another issue related to format recognition (GNU's tar(1) reads it successfully as well):

/home/am11/projects/consoletar/node-tar/test/fixtures/tars/next-file-has-long.tar ->

Unhandled Exception:
System.FormatException: Entry 'NextFileHasLongPath' was expected to be in the GNU format, but did not have the expected version data.
   at System.Formats.Tar.TarHeader.ReadVersionAttribute(Span`1 buffer) in /runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Read.cs:line 483
   at System.Formats.Tar.TarHeader.TryReadAttributes(TarEntryFormat iThe next initialFormat, Span`1 buffer) in /runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Read.cs:line 76
   at System.Formats.Tar.TarHeader.TryGetNextHeader(Stream archiveStream, Boolean copyData, TarEntryFormat initialFormat) in /runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Read.cs:line 31
   at System.Formats.Tar.TarReader.TryGetNextEntryHeader(Boolean copyData) in /runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarReader.cs:line 317
   at System.Formats.Tar.TarReader.GetNextEntry(Boolean copyData) in /runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarReader.cs:line 124
   at Program.<<Main>$>g__ReadTar|0_0(String path, Boolean useGz) in /home/am11/projects/consoletar/Program.cs:line 27
   at Program.<Main>$(String[] args) in /home/am11/projects/consoletar/Program.cs:line 15

There must be some way to gather a variety of them. Perhaps other tar implementations have a test bed we can reuse.

I was actually looking for implementation-agnostic spec tests suite for tar to evaluate TarReader's shortcomings etc., but couldn't find one (where we can insert our engine; as we did for System.Text.Json parsing with https://github.com/nst/JSONTestSuite).

Here are few related resources which can be adapted to run alternative parser:

• https://github.com/libarchive/libarchive/tree/master/libarchive/test - BSD
• https://github.com/npm/node-tar/tree/main/test - ISC
• https://github.com/landley/toybox/blob/master/tests/tar.test - 0BSD
• https://github.com/mirror/busybox/tree/master/testsuite/tar - GPLv2
• https://git.savannah.gnu.org/cgit/tar.git/tree/tests/testsuite.at - GPLv3

[danmoseley]

Thanks. I just now went through the Golang test tar files and found several more issues: #74316

[am11]

This delta fixes all issues with node-tar fixtures: main...am11:runtime:feature/system.formats.tar/hardlinks-support.

[danmoseley]

Nice! The 4 I had in #74316? Perhaps you could note that there.

We will need tests, which unfortunately may mean adding test .tar files to dotnet/runtime-assets, then letting it flow. If it's tested with a small change to the file, you could read an existing test tar into memory, reach into the buffer and twiddle some bits, then use TarReader on that, and avoid having to check in an asset.

[am11]

Sure closing this in favor of #74316.