DNS broken on ltsc2019 since the Sept 12 update

Describe the bug
Since the Sept 12 update for servercore:ltsc2019, DNS lookups always fail from within a container running that image.

To Reproduce
Steps to reproduce the behavior:

Run docker run mcr.microsoft.com/windows/servercore:ltsc2019 cmd /c ping www.microsoft.com
Actual result: Ping request could not find host www.microsoft.com. Please check the name and try again.

Expected behavior
DNS resolution for www.microsoft.com should succeed.

Configuration:

Edition: Windows Server 2019
Base Image being used: mcr.microsoft.com/windows/servercore:ltsc2019
Container engine: docker
Docker version 20.10.16, build 0744b98

Additional context
All lookups fail, not just www.microsoft.com. We have been able to work around the issue for now by forcing a different DNS server with a custom Docker network:

> docker network create --driver nat -o com.docker.network.windowsshim.dnsservers="8.8.8.8" natdnsworkaround
2ca5eeb673f4d0e18b1262c6d53794075c864280e4f87f928948e6d3f3b0738f
> docker run --network natdnsworkaround mcr.microsoft.com/windows/servercore:ltsc2019 cmd /c ping www.microsoft.com

Pinging e13678.dscb.akamaiedge.net [96.7.169.183] with 32 bytes of data: 
Reply from 96.7.169.183: bytes=32 time=25ms TTL=57 
Reply from 96.7.169.183: bytes=32 time=24ms TTL=57 
Reply from 96.7.169.183: bytes=32 time=24ms TTL=57 
Reply from 96.7.169.183: bytes=32 time=24ms TTL=57       

Ping statistics for 96.7.169.183:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), 
Approximate round trip times in milli-seconds:
    Minimum = 24ms, Maximum = 25ms, Average = 24ms

Mike Zappa (@MikeZappa87)

I think this looks similar to #216. Any thoughts on this Mike Zappa (@MikeZappa87) / kestratt?

We're also facing this problem, I've posted on the issue regarding the routing tables which is probably the same #386

Also facing the same issue. Getting failures when building docker images when a RUN directive tries to download from download.microsoft.com. The weird thing is that we started experiencing this on September 8th.

Workaround

I was able to workaround by setting the DNS server in the script being run using:

Get-NetAdapter | Get-DnsClientServerAddress -AddressFamily IPv4 | Set-DnsClientServerAddress -ServerAddresses ("8.8.8.8") -Validate

This is obviously not the way to go, especially since you have to run it in any layer that needs DNS...

any updates on this? this is blocker for all of our organizations trying to build their windows containers

Would also appreciate an update and a better workaround. This is a blocker for our customers.

Jeff Erbrecht (@jefferbrecht)

Jeff Erbrecht (@jefferbrecht), are you able to ping the dns server ip from the container? Could you also share the ipconfig output of in the container? -- docker run mcr.microsoft.com/windows/servercore:ltsc2019 cmd /c ipconfig /all

Unfortunately, I'm unable to reproduce this issue locally. The above repro steps are succeeding for me on server 2019 with KB5030214 installed. How exactly did you update your windows server?

I'm able to ping the workaround DNS server IP:

> docker run mcr.microsoft.com/windows/servercore:ltsc2019 cmd /c ping 8.8.8.8

Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=1ms TTL=117 
Reply from 8.8.8.8: bytes=32 time<1ms TTL=117 
Reply from 8.8.8.8: bytes=32 time<1ms TTL=117 
Reply from 8.8.8.8: bytes=32 time<1ms TTL=117 

Ping statistics for 8.8.8.8:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms

As for the default DNS servers, one works and one doesn't:

> docker run mcr.microsoft.com/windows/servercore:ltsc2019 powershell -command 'get-dnsclientserveraddress | ? { $_.AddressFamily -eq 2 }
 | % { $_.ServerAddresses | % { ping $_ } }'

Pinging 172.22.0.1 with 32 bytes of data: 
Reply from 172.22.0.1: bytes=32 time<1ms TTL=128
Reply from 172.22.0.1: bytes=32 time<1ms TTL=128 
Reply from 172.22.0.1: bytes=32 time<1ms TTL=128 
Reply from 172.22.0.1: bytes=32 time<1ms TTL=128 

Ping statistics for 172.22.0.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Pinging 169.254.169.254 with 32 bytes of data:
Request timed out. 
Request timed out. 
Request timed out. 
Request timed out. 

Ping statistics for 169.254.169.254:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

ipconfig output:

> docker run mcr.microsoft.com/windows/servercore:ltsc2019 cmd /c ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : xx
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid 
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : us-central1-a.c.xx.internal.

Ethernet adapter vEthernet (Ethernet):

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #2
   Physical Address. . . . . . . . . : xx-xx-xx-xx-xx-xx
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::6d22:7bbd:fc1e:xxxx%14(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.22.xx.xx(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.22.0.1
   NetBIOS over Tcpip. . . . . . . . : Disabled

I'm not sure I understand your last question. The server is a freshly created VM running on Google Compute Engine and I haven't intentionally done anything to it relating to updates. I can confirm that KB5030214 is installed on the server.

Jeff Erbrecht (@jefferbrecht)

Jeff Erbrecht (@jefferbrecht), Just to double check, is the 169.254.169.254 DNS server in a good state? Are you able to ping 169.254.169.254 from the host?

It's interesting that there isn't any DNS server info from the ipconfig /all output. Would you be able to run the following script on the host and share the output folder collectlogs.ps1?

https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/windows/debug/collectlogs.ps1

And you answered my update question. Thanks for confirming the KB was installed!

Yup, pinging from the host is working:

> ping 169.254.169.254

Pinging 169.254.169.254 with 32 bytes of data:
Reply from 169.254.169.254: bytes=32 time<1ms TTL=255
Reply from 169.254.169.254: bytes=32 time=1ms TTL=255
Reply from 169.254.169.254: bytes=32 time<1ms TTL=255
Reply from 169.254.169.254: bytes=32 time<1ms TTL=255

Ping statistics for 169.254.169.254:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms

Script output: output.zip

Let me add in, the problem is because the routing table inside the container doesn't work as expected, there is no active route for the default DNS server 169.254.169.254, please check this issue for more details #386 (comment)

Simply, run route print inside the container and you will find a persistent route to 169.254.169.254, not an active route, and without an active route it doesn't work correctly.

Samuel Almeida (@sam-sla)

Samuel Almeida (@sam-sla), I tried to use the fix advised in #386 (comment) and it works, but also doesn't persist between layers (e.g. if I build a container with these commands as a layer subsequent layers still fail to resolve names).

Any ideas on how to make these fixes persistent until the issue is resolved in the base image?

I am not sure, but it sounds like the route has to be fixed every time after the container is started.

What we ended up doing in our case as a workaround was to use a different DNS server that has a route, configured in the docker daemon, so it applies directly to every running container. https://learn.microsoft.com/en-us/virtualization/windowscontainers/manage-docker/configure-docker-daemon#configure-docker-with-a-configuration-file

I am having this issue as well, any idea when this might get fixed?

Same issue here, I would love to see this fixed.
For the time being I've added an init.ps1 script in the entrypoint that fires before starting the container service.
The script adds the route to the 169.254.169.254:

Write-Output "Add route to Google DNS"
route -p add 169.254.169.254 MASK 255.255.255.255 $($(Get-NetRoute | where {$_.DestinationPrefix -eq '0.0.0.0/0'}).NextHop)

Not a perfect solution but it does the work.

piomark

Same issue here, I would love to see this fixed. For the time being I've added an init.ps1 script in the entrypoint that fires before starting the container service. The script adds the route to the 169.254.169.254:
Write-Output "Add route to Google DNS"
route -p add 169.254.169.254 MASK 255.255.255.255 $($(Get-NetRoute | where {$_.DestinationPrefix -eq '0.0.0.0/0'}).NextHop)
Not a perfect solution but it does the work.

hi piomark! your workaround adds

ifIndex DestinationPrefix NextHop
17 169.254.169.254/32 172.21.224.1

but it doesn't really helps. for example, dist.nuget.org can't be resolved:

UnKnown can't find dist.nuget.org: Server failed
Server: UnKnown

is there any workaround without adding each servers' IPs to hosts?

Nurzhan Bazhikov (@nbazhikov-clgx)

Hi Nurzhan Bazhikov (@nbazhikov-clgx),

actually my example is for docker hosted on GCP VMs, 169.254.169.254 is internal google DNS. If your machine is hosted on other provider env you need to update the command accordingly. (you can check what is the host DNS server and configure the same for containers, you can check it with "ipconfig -all").
The command simply adds a route to DNS server via host.

Hope it helps

KB5030214 definitely did something bad to containers.

I have a pre-KB5030214 windowsservercore-2019 image and post-KB5030214 Windows 10 LTSC 2019 host system. Attempts to build any new image using docker build on top of that image results in error during layer commit:

re-exec error: exit status 1: output: hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)

Uninstalling KB5030214 fixes things back.

I decided to post it here so info about KB5030214 is kept together.

For anyone with this issue on GCP, we recently got the following from support:

"We understand that you're affected by the issue: GCE : DNS unresolved in Windows Container
. This issue was recently resolved by our Product Engineering Team. Please read below for updates on the status of your issue:

Status: Fully Resolved"

and

"Late August Google made a change to the underlying DHCP service used by the GCP infrastructure which impacted Windows Containers. The reason this bug was overlooked was because there was no or insufficient test coverage for Windows Containers whose network configuration is different than that of the Windows host OS. In any case, I've been made aware that the code that caused the regression has already been rolled back.

Last month Google engineering migrated to a new DHCP service that configured all VMs (including Windows VMs) to receive the IP address 169.254.169.254 as a DNS server IP address. In Windows, this is type of IP address is called Automatic Private IP Addressing (APIPA) which is considered a local link and should not be used for other purposes, such as DNS. The correct DNS configuration is to use the subnet's default gateway as the DNS server IP address. [1]

Despite this misconfiguration the IP address is actually still routable on the host VM, hence hostname resolution was still working. However, when running containers the invalid VM DNS configuration is automatically carried over to the container layer where there is another level of abstraction. By design [2] local links should not be able to route this IP address which the container correctly enforced. Prior to the bug the containers correctly received the default subnet gateway of the VM as the container's DNS IP address and therefore resolved hostnames as normal.

As an action item to prevent this type of issue from occurring again Cloud Support will be asking Google engineering to ensure full feature parity when migrating from legacy to new services. In this instance, the engineering team must expand test coverage to ensure that the network configuration of Windows VMs is properly configured by the DHCP service to use the default gateway IP address as the target DNS server.

Marat Radchenko (@slonopotamus)

I decided to post it here so info about KB5030214 is kept together.

Marat Radchenko (@slonopotamus): looks like this one was a different issue, and should probably be opened as a new ticket to track is. This ticket turns out to be a Google-side misconfiguration, and can probably be closed on that basis.

Yep, I understand.

Docker image ltsc2019 has been updated. This issue fixed in my environments.

Digest:sha256:b3b5f1d43d7004541aae6f0c2ead0c70d943c85653f0e37861c73eaa9d687c42
Last Modified 10/11/2023

docker pull mcr.microsoft.com/windows/servercore:ltsc2019

https://mcr.microsoft.com/en-us/product/windows/servercore/tags

Can also confirm that the issue has been resolved in my environment.

Can also confirm here. Thanks!

Navigation Menu

Repository navigation

DNS broken on ltsc2019 since the Sept 12 update #420

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DNS broken on ltsc2019 since the Sept 12 update #420

Description

Activity

fady-azmy-msft commented on Sep 14, 2023

sam-sla commented on Sep 15, 2023

danaradg commented on Sep 18, 2023

Workaround

nbazhikov-clgx commented on Sep 25, 2023

danaradg commented on Sep 25, 2023

kestratt commented on Sep 26, 2023

jefferbrecht commented on Sep 26, 2023

kestratt commented on Sep 26, 2023

jefferbrecht commented on Sep 26, 2023

sam-sla commented on Sep 27, 2023

danaradg commented on Sep 27, 2023

sam-sla commented on Sep 28, 2023

hickert33 commented on Sep 28, 2023

piomark commented on Sep 29, 2023

nbazhikov-clgx commented on Oct 2, 2023

piomark commented on Oct 3, 2023

slonopotamus commented on Oct 3, 2023

khenkel commented on Oct 10, 2023

TBBle commented on Oct 10, 2023

slonopotamus commented on Oct 10, 2023

sumof2primes commented on Oct 11, 2023

danaradg commented on Oct 11, 2023

jefferbrecht commented on Oct 13, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions