runtime-extra-platforms rolling build failure: Docker Internal Server Error - The container operating system does not match the host operating system.

[carlossanlop]

Seen in the two latest rolling builds from runtime-extra-platforms:

• 309 errors: https://dev.azure.com/dnceng/public/_build/results?buildId=1703729&view=results
• 312 errors: https://dev.azure.com/dnceng/public/_build/results?buildId=1704666&view=results

Both rolling builds had a general problem causing most tests to output this to the log:

Console log: 'Microsoft.Extensions.Configuration.Xml.Tests' from job 6602cf93-0be3-4663-a60d-e81987ec3e49 (windows.amd64.server2022.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-2004-helix-amd64-20200904200251-272704c on a001B54
running %HELIX_CORRELATION_PAYLOAD%\scripts\414866fc2fa14b85a4c4464ddf625f32\execute.cmd in C:\h\w\A56B0923\w\ABBF0971\e max 900 seconds

Output:

Exit Code:-900

The run_client.py log from artifacts show this:

2022-04-06T22:48:20.058Z	ERROR  	dockerhelper(233)	run	API Error attempting to talk to Docker Server:500 Server Error: Internal Server Error ("CreateComputeSystem 9098476f85a32e792494273ed6ffbdda89e2eaa58e6afaf0e6078dc89ebefbf4: The container operating system does not match the host operating system.
(extra info: {"SystemType":"Container","Name":"9098476f85a32e792494273ed6ffbdda89e2eaa58e6afaf0e6078dc89ebefbf4","Owner":"docker","VolumePath":"\\\\?\\Volume{657b1847-3ff3-4c95-89f0-1b5e5411847e}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\9098476f85a32e792494273ed6ffbdda89e2eaa58e6afaf0e6078dc89ebefbf4","Layers":[{"ID":"274808d8-c4a3-528d-bf38-5bc4d461b846","Path":"C:\\ProgramData\\docker\\windowsfilter\\072f4aae481ffe6f2d48c9a39da42b0b39d1574d23b4a4fc82dc44acb2e9de2b"},{"ID":"99645de1-1409-5d0a-b552-4e1371e5a60e","Path":"C:\\ProgramData\\docker\\windowsfilter\\70f9fcabe746be1c425e0ffba76d8b982c1cdc8d2737cd60adc7fc1aaaa70d30"},{"ID":"7c6c0748-5ffd-5279-b129-e3c5ae923d8e","Path":"C:\\ProgramData\\docker\\windowsfilter\\c5d218a45573c713adfd6bbc30e7f06d517d3fcd04753f94cd7519af80774603"},{"ID":"b93eb741-0f60-5b94-8bb5-8ac866f67937","Path":"C:\\ProgramData\\docker\\windowsfilter\\390e650a559fcbf01ea9b6a2c0ab9c995e40c5bc3649ef7be740fc141d501baf"},{"ID":"aace227d-c83e-5055-9c4a-75cc06886df6","Path":"C:\\ProgramData\\docker\\windowsfilter\\80ae038a6f851f11826739f7c4eab1b25554a93d6294ad507429fd3ec2bea6cc"},{"ID":"4c612d25-6193-576e-b50c-4f2a075b4061","Path":"C:\\ProgramData\\docker\\windowsfilter\\e2c8aa54c24a31681076052d879682b5eb69c1b39b2a4455ffee9eedd6a9d4ac"}],"HostName":"Da001B53","MappedDirectories":[{"HostPath":"c:\\users\\runner\\appdata\\local\\temp\\66492a4b73304872927afce22db1de83","ContainerPath":"c:\\commands","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\scripts\\helix-scripts-no-deps","ContainerPath":"c:\\helix\\scripts","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\cores","ContainerPath":"c:\\helix\\cores","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\logs","ContainerPath":"c:\\helix\\logs","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\config","ContainerPath":"c:\\helix\\config","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\p","ContainerPath":"c:\\helix\\work\\correlation","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\w\\b7140a28\\u","ContainerPath":"c:\\helix\\work\\workitem\\u","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\w\\b7140a28\\e","ContainerPath":"c:\\helix\\work\\workitem\\e","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\w\\b7140a28\\uploads","ContainerPath":"c:\\helix\\work\\workitem\\uploads","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false}],"HvPartition":false,"EndpointList":["5640acae-c836-4c9d-9308-33ce846942cf"],"AllowUnqualifiedDNSQuery":true})")
2022-04-06T22:48:20.060Z	ERROR  	dockerhelper(234)	run	Docker may not be configured correctly on this machine.  Contact dnceng for help.

@MattGal @danmoseley @ilyas1974

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Seen in the two latest rolling builds from runtime-extra-platforms:

• 309 errors: https://dev.azure.com/dnceng/public/_build/results?buildId=1703729&view=results
• 312 errors: https://dev.azure.com/dnceng/public/_build/results?buildId=1704666&view=results

Both rolling builds had a general problem causing most tests to output this to the log:

Console log: 'Microsoft.Extensions.Configuration.Xml.Tests' from job 6602cf93-0be3-4663-a60d-e81987ec3e49 (windows.amd64.server2022.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-2004-helix-amd64-20200904200251-272704c on a001B54
running %HELIX_CORRELATION_PAYLOAD%\scripts\414866fc2fa14b85a4c4464ddf625f32\execute.cmd in C:\h\w\A56B0923\w\ABBF0971\e max 900 seconds

Output:

Exit Code:-900

The run_client.py log from artifacts show this:

2022-04-06T22:48:20.058Z	ERROR  	dockerhelper(233)	run	API Error attempting to talk to Docker Server:500 Server Error: Internal Server Error ("CreateComputeSystem 9098476f85a32e792494273ed6ffbdda89e2eaa58e6afaf0e6078dc89ebefbf4: The container operating system does not match the host operating system.
(extra info: {"SystemType":"Container","Name":"9098476f85a32e792494273ed6ffbdda89e2eaa58e6afaf0e6078dc89ebefbf4","Owner":"docker","VolumePath":"\\\\?\\Volume{657b1847-3ff3-4c95-89f0-1b5e5411847e}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\9098476f85a32e792494273ed6ffbdda89e2eaa58e6afaf0e6078dc89ebefbf4","Layers":[{"ID":"274808d8-c4a3-528d-bf38-5bc4d461b846","Path":"C:\\ProgramData\\docker\\windowsfilter\\072f4aae481ffe6f2d48c9a39da42b0b39d1574d23b4a4fc82dc44acb2e9de2b"},{"ID":"99645de1-1409-5d0a-b552-4e1371e5a60e","Path":"C:\\ProgramData\\docker\\windowsfilter\\70f9fcabe746be1c425e0ffba76d8b982c1cdc8d2737cd60adc7fc1aaaa70d30"},{"ID":"7c6c0748-5ffd-5279-b129-e3c5ae923d8e","Path":"C:\\ProgramData\\docker\\windowsfilter\\c5d218a45573c713adfd6bbc30e7f06d517d3fcd04753f94cd7519af80774603"},{"ID":"b93eb741-0f60-5b94-8bb5-8ac866f67937","Path":"C:\\ProgramData\\docker\\windowsfilter\\390e650a559fcbf01ea9b6a2c0ab9c995e40c5bc3649ef7be740fc141d501baf"},{"ID":"aace227d-c83e-5055-9c4a-75cc06886df6","Path":"C:\\ProgramData\\docker\\windowsfilter\\80ae038a6f851f11826739f7c4eab1b25554a93d6294ad507429fd3ec2bea6cc"},{"ID":"4c612d25-6193-576e-b50c-4f2a075b4061","Path":"C:\\ProgramData\\docker\\windowsfilter\\e2c8aa54c24a31681076052d879682b5eb69c1b39b2a4455ffee9eedd6a9d4ac"}],"HostName":"Da001B53","MappedDirectories":[{"HostPath":"c:\\users\\runner\\appdata\\local\\temp\\66492a4b73304872927afce22db1de83","ContainerPath":"c:\\commands","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\scripts\\helix-scripts-no-deps","ContainerPath":"c:\\helix\\scripts","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\cores","ContainerPath":"c:\\helix\\cores","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\logs","ContainerPath":"c:\\helix\\logs","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\config","ContainerPath":"c:\\helix\\config","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\p","ContainerPath":"c:\\helix\\work\\correlation","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\w\\b7140a28\\u","ContainerPath":"c:\\helix\\work\\workitem\\u","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\w\\b7140a28\\e","ContainerPath":"c:\\helix\\work\\workitem\\e","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false},{"HostPath":"c:\\h\\w\\a56b0923\\w\\b7140a28\\uploads","ContainerPath":"c:\\helix\\work\\workitem\\uploads","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false}],"HvPartition":false,"EndpointList":["5640acae-c836-4c9d-9308-33ce846942cf"],"AllowUnqualifiedDNSQuery":true})")
2022-04-06T22:48:20.060Z	ERROR  	dockerhelper(234)	run	Docker may not be configured correctly on this machine.  Contact dnceng for help.

@MattGal @danmoseley @ilyas1974

Author:	carlossanlop
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure, untriaged
Milestone:	-

[ilyas1974]

To keep you in the loop, this is what I found - "It depends on the isolation type being used. On Windows Server, process isolation is used by default. With process isolation, the host and container must be the same version. With Hyper-V isolation, the container can be the same version or a lower version than the host. This is described at https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility with some helpful tables."

I've created dotnet/arcade#8981 to investigate

[carlossanlop]

Continues affecting the runtime-extra-platforms rolling builds from 2022/04/08:

308 failures: https://dev.azure.com/dnceng/public/_build/results?buildId=1706951&view=results
310 failures: https://dev.azure.com/dnceng/public/_build/results?buildId=1705858&view=results

[carlossanlop]

Continues happening on runtime-extra-platforms consistently:

• 04/10 (first rolling build) 331 failures: https://dev.azure.com/dnceng/public/_build/results?buildId=1709605&view=results
• 04/10 (second rolling build) 348 failures: https://dev.azure.com/dnceng/public/_build/results?buildId=1709911&view=results
• 04/11: 329 failures: https://dev.azure.com/dnceng/public/_build/results?buildId=1710404&view=results

[AaronRobinsonMSFT]

Seems to be occurring on #68615 regularly.

[elinor-fung]

Note that #68615 failures were not on Windows Server 2022:

Job        : b493b6c6-de4b-452c-95a1-8639afa964e1
QueueAlias : windows.nano.1809.amd64.open
QueueName  : windows.10.amd64.serverrs5.open.rt
DockerTag  : mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504

[MattGal]

Note that #68615 failures were not on Windows Server 2022:

Job        : b493b6c6-de4b-452c-95a1-8639afa964e1
QueueAlias : windows.nano.1809.amd64.open
QueueName  : windows.10.amd64.serverrs5.open.rt
DockerTag  : mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504

I think we have a bug from the other attempted fix; I'm trying some stuff and will reply back shortly.

[MattGal]

Rolling back the change that added hyper-v isolation calls seems to have unblocked this.

[jakobbotsch]

Closing as per above.