GetFileSystemEntries() causes each file to be stat'd even if the only thing we need are the paths

[antiufo]

Original issue: PowerShell/PowerShell#10816.
It seems like the root cause is that string[] IO.Directory.GetFileSystemEntries() calls stat() on each entry to retrieve additional metadata, even though the only thing that method is requesting is the path, which should readily be available in the data provided by readdir().

This can be observed by running strace -f -tt -o trace.txt pwsh -noprofile -c '$a = [IO.Directory]::GetFileSystemEntries("/usr/bin")'

I quickly looked at the code and it seems like additional metadata should be retrieved lazily in FileSystemEntry.Unix.cs/FileStatus.Unix.cs (which is a reasonable thing to do), but I could be wrong. Is there something that is unexpectedly triggering stat()?

[wfurt]

Problem with lazy init is that you may not get consistent info or it may not work if something changes.
cc: @JeremyKuhne

[JeremyKuhne]

@NattyNarwhal this appears to be a regression from dotnet/corefx#34088. Can you take a look?

[NattyNarwhal]

I'm not entirely sure about that. That patch basically does another check if the dirent is DT_UNKNOWN (since some systems, ahem may not return type info in a dirent), but the code was already doing a stat before anyways (to find the true nature, be it because its a symlink to resolve or an unknown as sentinel) - I just made it do it again if DT_UNKNOWN to check if its a symlink.

[JeremyKuhne]

I'm not entirely sure about that.

Sorry, I think you're right- reviewing a lot of System.IO issues. :)

@antiufo The only time we should be doing stat is for links and unknown types. Are you seeing something to the contrary? We could consider making lazily doing that an option, but we wouldn't want to do it by default.

[antiufo]

@antiufo The only time we should be doing stat is for links and unknown types.

With many links in /usr/bin to remote drives, as a Googler, this effectively prevents me from using PowerShell as my main shell (due to the unpredictable random delays).

We could consider making lazily doing that an option, but we wouldn't want to do it by default.

May I ask why? Does this mean that even a string[] GetFileSystemEntries() will potentially stat()?

[markmcgookin]

Just something for someone else that eventually stumbles across this... I was using GetFileSystemEntries to iterate over secrets provided by the Azure Keyvault Secret provider for K8S (which mounts secrets as files in a volume) and after an update I was getting "Unhandled exception. System.UnauthorizedAccessException: Access to the path '/mnt/secrets-store/..data' is denied."

I don't know enough about what was going on under the hood or what the ..data suffix is there... but I was having issues when using the mcr.microsoft.com/dotnet/aspnet:6.0.2-bullseye-slim image.

A workaround I have found was to use Directory.GetFiles instead of GetFileSystemEntries and it works. I wonder is that particular image denying access because of the stat() call.

Anyway... I ended up here after a lot of googling, maybe someone else will too and this will help.

[adamsitnik]

@tmds is it still a thing after the refactor that you have made few months ago?