Validate container improvements with .NET 6

[richlander]

Validate container improvements with .NET 6

We've made various changes in .NET 6 to improve aspects of container resource limits governance, particularly for Windows containers. This issue demonstrates the new capabilities and their current behavior using the dotnetapp sample app.

Issues:

• DOTNET_PROCESSOR_COUNT is a hex value -- should be decimal
• Parse DOTNET_PROCESSOR_COUNT with a 10 radix not 16

Changes:

• Container limits not supported by .NET runtime in process isolation mode -- Windows containers
• Enable Environment.ProcessorCount to return total machine processors in containers
• Add use of GCMemoryInfo.TotalAvailableMemoryBytes

The last change is to a sample, which will be used to demonstrate the others changes. It is a .NET 5 app. As a result, various roll-forward settings are used to coerce the app to run on .NET 6.

At the time of writing, the relevant changes have not been released in a public .NET 6 preview. As a result, I updated and locally rebuilt various Dockerfiles with daily builds from dotnet/installer.

Relevant docs:

• Testing GC Heap Counts with Containers #71413
• .NET Core GC Support for Docker Limits
• docker run -- runtime constraints on resources
• Windows Container Isolation Modes

Linux -- CPU and memory limits

The following test demonstrates the .NET runtime honoring CPU and resource limits.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.2.21154.6
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 3
TotalAvailableMemoryBytes: 75.00 MiB
cfs_quota_us: 300000
usage_in_bytes: 25821184 24.63 MiB
limit_in_bytes: 104857600 100.00 MiB

Similar, but with CPU sets (AKA CPU affinity) is used instead.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus "1-3,5" -m 100mb -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.2.21154.6
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 4
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 6459392 6.16 MiB
limit_in_bytes: 104857600 100.00 MiB

The latest .NET 6 Preview is used for these examples, since these capabilities are all pre .NET 6.

Linux -- custom processor count

The following test demonstrates the .NET runtime honoring the new DOTNET_PROCESSOR_COUNT ENV, which provides a different value to Environment.ProcessorCount and the equivalent setting in the native runtime. It doesn't directly related to and does not affect the docker run --cpus mechanism, but is intended as a higher-level signal for scaling algorithms.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
cfs_quota_us: 300000
usage_in_bytes: 6885376 6.57 MiB
limit_in_bytes: 104857600 100.00 MiB

I used a .NET 6 Preview 6 build to demonstrate this change. It works as expected.

Let's double check that we get the right behavior if no CPU limits are set.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 6713344 6.40 MiB
limit_in_bytes: 104857600 100.00 MiB

We do.

Let's check if we get the correct value with CPU affinity.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus "1-3,5" -m 100mb -e DOTNET_PROCESSOR_COUNT=10 -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:/app -w /app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Debian GNU/Linux bullseye/sid

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB
usage_in_bytes: 5898240 5.63 MiB
limit_in_bytes: 104857600 100.00 MiB

We do.

Windows -- CPU and memory limits

The major container improvement changes were made on Windows in .NET 6. Let's try with the .NET 6 preview 3. We're expecting that CPU limits are not honored with process isolated containers.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor mcr.microsoft.com/dotnet/runtime:6.0 dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.3.21201.4
Microsoft Windows 10.0.19042

OSArchitecture: X64
ProcessorCount: 16
TotalAvailableMemoryBytes: 75.00 MiB

As expected, processor count is incorrect.

Note: You will see the correct behavior with Hyper-V isolated containers. The functionality gap is with process-isolated containers.

Let's try .NET 6 Preview 6.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21272.4
Microsoft Windows 10.0.19042

OSArchitecture: X64
ProcessorCount: 3
TotalAvailableMemoryBytes: 75.00 MiB

Perfect.

Now let's try CPU affinity, like we did with Linux.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpuset-cpus="1-3" -m 100mb --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor -e DOTNET_PROCESSOR_COUNT=10 dotnet6runtime dotnet dotnetapp.dll
docker: Error response from daemon: invalid option: Windows does not support CpusetCpus.
See 'docker run --help'.

It isn't supported. I didn't know that.

Windows -- custom processor count

Just like with Linux, let's try setting a custom processor count.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm --cpus 3 -m 100mb -e DOTNET_PROCESSOR_COUNT=10 --isolation=process -v C:\git\dotnet-docker\samples\dotnetapp\bin\Debug\net5.0:C:\app -w C:\app -e DOTNET_ROLL_FORWARD=LatestMajor dotnet6runtime dotnet dotnetapp.dll
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.0-preview.6.21276.13
Microsoft Windows 10.0.19042

OSArchitecture: X64
ProcessorCount: 10
TotalAvailableMemoryBytes: 75.00 MiB

As expected, it behaves just like demonstrated with Linux.

Validating isolation mode

Here's what I did to validate the isolation mode.

C:\git\dotnet-docker\samples\dotnetapp>docker run --rm -it --isolation=process --name dotnet6runtime dotnet6runtime

And then in another command prompt.

C:\>docker ps
CONTAINER ID   IMAGE            COMMAND                    CREATED          STATUS          PORTS     NAMES
ee11156d0488   dotnet6runtime   "c:\\windows\\system32…"   15 seconds ago   Up 14 seconds             dotnet6runtime

C:\>docker inspect dotnet6runtime | findstr Isolation
            "Isolation": "process",

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jeffschwMSFT]

cc @mangod9

[richlander]

I updated the testing data above with the latest build. Everything looks great. Thanks!

@AntonLapounov @AaronRobinsonMSFT @jkotas

This issue can be closed at any time.

[mangod9]

Nice.

[richlander]

Similar topic for .NET Framework: microsoft/dotnet-framework-docker#935