feat(discv5)!: prefer IPv4 for dual-stack ENRs and fail requests on send error - #341
Open
MysticRyuujin wants to merge 1 commit into
Open
MysticRyuujin wants to merge 1 commit into
MysticRyuujin wants to merge 1 commit into
Conversation
…end error Dual-stack ENRs are contacted over IPv4 first (configurable via dualStackPreferredFamily). UDP send errors fail the request at once with SendFailed. The address a node last sent an authenticated message from is reused for later requests. The IPv6 vote-harvest ping is an isolated probe. verifyEnr accepts a dual-stack ENR observed over either family.
Merged
1 task done
nflaig
added a commit
to ChainSafe/lodestar
that referenced
this pull request
Sep 19, 2026
…10104) ## Motivation A fresh v1.48.0 beacon node on sepolia or hoodi that runs in a Docker container without an IPv6 route finds zero peers. Discovery never sends a UDP packet. `parseListenArgs` binds `::` when the user sets neither `--listenAddress` nor `--listenAddress6`. With an IPv6 socket bound, `@chainsafe/discv5` contacts every dual-stack ENR over IPv6 and does not fall back to IPv4 (`getSocketAddressMultiaddrOnENR`, same rule as sigp/discv5 `DualStack`). In a container without IPv6 connectivity every send to those peers fails. Until v1.47.0 the built-in sepolia and hoodi bootnode lists contained IPv4-only EF bootnodes, which masked this. #10049 replaced them with the NodeOps fleet, which is dual-stack. The only IPv4-only entry left (a Teku node) does not answer. The v1.48.0 lists therefore contain no bootnode a default-configured IPv4-only container can reach. Mainnet still has IPv4-only bootnodes and is not affected. Verified on a v4-only Docker host on sepolia with the fleet ENRs as `--bootnodes`: | image | flags | result after 90s | | --- | --- | --- | | v1.48.0 | default | 0 peers, 0 ENRs decoded, 0 UDP packets sent | | v1.48.0 | `--listenAddress 0.0.0.0` | syncing, hundreds of ENRs discovered | | v1.48.0 | `--bootnodes <IPv4-only EF ENR>` | syncing | | v1.47.0 | default | syncing (built-in IPv4-only EF bootnodes) | ## Description Bind IPv6 by default only if the host has a global unicast IPv6 address. `hasGlobalIPv6Address` reads `os.networkInterfaces()` and accepts only addresses inside the IANA global unicast block `2000::/3`, minus the special-purpose prefixes allocated inside it that the IANA registry marks not globally reachable (IETF protocol assignments including Teredo, 6to4, documentation, SRv6 SIDs), through two `net.BlockList`s. Everything else (loopback, IPv4-mapped, NAT64, unique-local, link-local, site-local) falls outside `2000::/3`. An explicit `--listenAddress6` still forces IPv6 on. An explicit `--listenAddress` still turns the IPv6 default off, as before. `parseListenArgs` takes the detection result as a second parameter so tests stay independent of the test host. Adds unit tests, updates the `--listenAddress6` description and the networking docs. Validated with an image built from this branch on the same IPv4-only Docker host: default flags bind IPv4 only (`bindAddr4=/ip4/0.0.0.0/udp/...`, no `bindAddr6`) and discover peers on sepolia and hoodi; `--listenAddress6 ::` still binds IPv6 as before. Lodestar warns at startup when it skips the IPv6 default for this reason. A persisted ENR loses its `ip6`/`udp6`/`tcp6`/`quic6` fields when no IPv6 listener is configured and no `--enr.*6` flag is given, so peers stop dialing a dead IPv6 endpoint and the beacon handler's `enrUpdate` gate is no longer blocked by a stale `ip6`. The library-side fix, IPv4-first selection and fast failure on send errors in `@chainsafe/discv5`, is ChainSafe/discv5#341. ## AI Assistance Disclosure - [x] External Contributors: I have read the [contributor guidelines](https://github.com/ChainSafe/lodestar/blob/unstable/CONTRIBUTING.md#ai-assistance-notice) and disclosed my usage of AI below. This PR was written primarily by Claude Code, including the investigation, the code, the tests and this description. I directed the work and validated the fix on our own infrastructure. A second model (Codex) reviewed the diff before submission. Replies in this PR may also be drafted with AI assistance. --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A node that binds both
0.0.0.0and::contacts every dual-stack ENR over IPv6 and never falls back.getSocketAddressMultiaddrOnENRprefersudp6wheneveripMode.ip6is set, the chosen address is frozen into theNodeContact, andtransport/udp.tscallssocket.sendwithout a callback, so a failed send is never observed. WithrequestRetries: 1the request dies on the 1 s timeout with zero retransmits.In a Docker container without an IPv6 route this means zero peers forever once the bootnode list has no IPv4-only entry. Lodestar v1.48.0 hit exactly this on sepolia and hoodi after the built-in bootnodes moved to a dual-stack fleet (ChainSafe/lodestar#10104 is the caller-side mitigation). Fresh nodes sent no discv5 packets at all; the same image with an IPv4-only bind discovered peers at once.
A second defect surfaced while tracing this:
verifyEnrjoins the family comparisons with??, so for a dual-stack ENR observed over IPv6 the IPv4 comparison returnsfalseand the IPv6 endpoint is never checked. With the defaultallowUnverifiedSessions: falsethe session is dropped. Dual-stack peers that reach a dual-stack node over IPv6 are rejected today.Description
ISessionConfig.dualStackPreferredFamily(4 | 6, default4).IPModestays a pure capability. This matches go-ethereum, which prefers IPv4 on a tie for dual-stack records.UDPTransportService.sendresolves or rejects through thesocket.sendcallback. dgram reports send errors only that way. A rejected request send fails the request at once with the newRequestErrorType.SendFailed, guarded against late callbacks for a request that was already answered or replaced. Response and challenge sends keep the log-only path.Discv5remembers the socket address each node last sent an authenticated message from (confirmedAddrs, written on decrypted requests and verified responses only, bounded like the session cache, cleared onDisconnectedandstop). Later requests to that node reuse it throughcontactFor, so a peer that dialed us over IPv6 keeps getting IPv6 and IPv6 address votes keep flowing under the IPv4-first default. An outgoing handshake never writes the cache: it proves nothing about the address until something comes back.createNodeContactAtbuilds anEnrcontact at a given bare address; aRawcontact would trigger the internal FINDNODE(0) ENR exchange after session expiry and carry a/p2psuffix that fails the reply address check.failRequesthands the queue tosendNextRequestinstead offailSession) and does not mark the peerDisconnectedinrpcFailure. The probe flag survives the pending queue.verifyEnraccepts a dual-stack ENR observed over either family.discv5_request_failed_count{reason}.Breaking
Dual-stack peers are now contacted over IPv4 first. Set
dualStackPreferredFamily: 6to restore the previous order. A host with working IPv6 and broken IPv4 loses connectivity to dual-stack peers until it sets that option; a follow-up adds per-request fallback to the other family.Tests
test/unit/util/ip.test.ts,test/unit/session/nodeInfo.test.ts: selection order and the endpoint-override contact.test/unit/transport/udp.test.ts:sendrejects on an undeliverable datagram, before start, and for an unbound family.test/unit/session/service.test.ts: stub transport;SendFailedwithout waiting for the timeout, queue failure semantics, probe isolation, probe flag through the queue,verifyEnrover either family.test/e2e/dualStack.test.ts: loopback nodes where the peer advertises an unreachable IPv6 address next to reachable127.0.0.1. Default reaches it over IPv4 (2001:db8::1, which some kernels blackhole and others refuse); IPv6-first fails fast withSendFailed(::ffff:127.0.0.1, which an IPv6-only socket refuses on every platform); a peer that dialed us over::1is contacted back over::1.The existing
mainnetBootnodese2e cannot show this regression because the mainnet list still has IPv4-only entries.Follow-up
Sequential endpoint fallback in
SessionService: try the other family after aSendFailedor a timeout, with a bounded success cache. This PR leaves the single request-send chokepoint, theprobeflag in both layers andcontactForin place for it.Verification on an IPv4-only Docker host
Lodestar built from a throwaway branch that overrides
@chainsafe/discv5with this branch, run on a Linux host whose Docker bridge has no IPv6 route, with--listenAddress 0.0.0.0 --listenAddress6 ::so both sockets are bound, built-in bootnodes only (SKIP_FETCH_NETWORK_BOOTNODES=1), 90 seconds after start:The unpatched node never decodes a single ENR: every request goes to an IPv6 address and dies on the timeout. The
SendFailedcount on the patched node is the IPv6 vote-harvest probes, which are isolated from the routing table by design. The unit and e2e suites also pass on that host inside anode:24container both with host networking (IPv6 route present) and on the bridge network (none).AI Assistance Disclosure
This PR was written primarily by Claude Code: the investigation, the code, the tests and this description. I directed the work and validated the fix on our own infrastructure. A second model (Codex) reviewed the design twice and the diff twice before submission. Replies in this PR may also be drafted with AI assistance.