cli/command: use longer init-ping timeout for ssh connections - #7337
Open
adisivaprasad wants to merge 1 commit into
Open
adisivaprasad wants to merge 1 commit into
adisivaprasad wants to merge 1 commit into
Conversation
The CLI performs an initial ping to the daemon during (lazy)
initialization, which is used for API-version negotiation and to
collect server info (e.g. whether experimental features are enabled,
and the daemon's OS type). This ping has a short (2 second) timeout
to prevent the CLI from hanging when connecting to a local daemon
that is not running.
Establishing a connection through the ssh connection helper requires
a TCP connection, host key verification, and authentication before
the connection to the daemon is ready to be used, which can take
longer than this timeout, in particular when connecting to a host
for the first time. In that case, the initial ping is aborted and
API-version negotiation is skipped: the client uses its default
(maximum) API version, and requests to a daemon with a lower API
version fail with errors such as:
Error response from daemon: client version 1.56 is too new.
Maximum supported API version is 1.55
This can be observed with 'docker --debug --context <ssh-context>
version', which dials the daemon twice on a slow (first) connection,
and prints the client's default (maximum) API version instead of the
negotiated version.
The ssh connection helper sets a default ConnectTimeout=30 for the
ssh process, so allow the initial ping for ssh connections to complete
within that time, plus some additional time for the handshake and the
ping itself. Non-ssh connections (local sockets, tcp) keep the default
(short) timeout, as those connections are either established quickly,
or fail quickly when the daemon is not running.
Fixes docker#6125
Signed-off-by: adisivaprasad <adisivaprasad@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
- What I did
Fixes #6125
The CLI performs an initial ping to the daemon during (lazy) initialization, which is used for API-version negotiation and to collect server info (e.g. whether experimental features are enabled, and the daemon's OS type). This ping has a short (2 second) timeout to prevent the CLI from hanging when connecting to a local daemon that is not running (#3652, #4226).
Establishing a connection through the ssh connection helper requires a TCP connection, host key verification, and authentication before the connection to the daemon is ready to be used, which can take longer than this timeout, in particular when connecting to a host for the first time. In that case the initial ping is aborted, and API-version negotiation is skipped: the client keeps its default (maximum) API version, and requests to a daemon with a lower API version fail with errors such as:
This can be observed with
docker --debug --context <ssh-context> version, which dials the daemon twice on a slow (first) connection (the first connection is killed when the ping's 2-second deadline expires), and prints the client's default (maximum) API version instead of the negotiated version:- How I did it
The ssh connection helper sets a default
ConnectTimeout=30for the ssh process (cli/connhelper), so this change allows the initial ping forssh://endpoints to complete within that time, plus some additional time for the handshake and the ping itself (sshInitTimeout = 32s). Non-ssh connections (local sockets, tcp) keep the default (short) timeout, as those connections are either established quickly, or fail quickly when the daemon is not running, so their fast-fail behavior is unchanged.- How I verified it
Reproduced the issue by simulating a slow ssh connection (an
sshwrapper that delays the connection by ~3 seconds, i.e. longer thandefaultInitTimeout) against a local daemon with a lower API version than the client:Before (client built from master):
After:
The initial ping now completes within the longer timeout, the connection is reused for the subsequent request (only one ssh connection is started), and the API version is negotiated correctly.
Also added unit tests:
TestGetInitTimeout: verifies the timeout selection per endpoint type (default for unix/tcp, longer for ssh, explicitinitTimeoutoverride takes precedence for both).TestInitializeFromClientSlowConnection: simulates a daemon that is slow to connect to (dial takes longer thandefaultInitTimeout), and verifies that for anssh://endpoint the initial ping completes and the API version is negotiated, whereas non-ssh endpoints keep the default fast-fail behavior.go test ./cli/command/...passes, andgolangci-lint(v2.10.1, as used in CI) reports no issues.