Skip to content

cli/command: use longer init-ping timeout for ssh connections - #7337

Open
adisivaprasad wants to merge 1 commit into
docker:masterfrom
adisivaprasad:ssh-init-ping-timeout
Open

adisivaprasad wants to merge 1 commit into
docker:masterfrom
adisivaprasad:ssh-init-ping-timeout

Conversation

@adisivaprasad

Copy link
Copy Markdown

- What I did

Fixes #6125

The CLI performs an initial ping to the daemon during (lazy) initialization, which is used for API-version negotiation and to collect server info (e.g. whether experimental features are enabled, and the daemon's OS type). This ping has a short (2 second) timeout to prevent the CLI from hanging when connecting to a local daemon that is not running (#3652, #4226).

Establishing a connection through the ssh connection helper requires a TCP connection, host key verification, and authentication before the connection to the daemon is ready to be used, which can take longer than this timeout, in particular when connecting to a host for the first time. In that case the initial ping is aborted, and API-version negotiation is skipped: the client keeps its default (maximum) API version, and requests to a daemon with a lower API version fail with errors such as:

Error response from daemon: client version 1.56 is too new. Maximum supported API version is 1.55

This can be observed with docker --debug --context <ssh-context> version, which dials the daemon twice on a slow (first) connection (the first connection is killed when the ping's 2-second deadline expires), and prints the client's default (maximum) API version instead of the negotiated version:

$ DOCKER_HOST=ssh://remote docker --debug version
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- remote docker system dial-stdio]"
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- remote docker system dial-stdio]"
Client:
 Version:           29.8.1
 API version:       1.56      <- not negotiated
...

- How I did it

The ssh connection helper sets a default ConnectTimeout=30 for the ssh process (cli/connhelper), so this change allows the initial ping for ssh:// endpoints to complete within that time, plus some additional time for the handshake and the ping itself (sshInitTimeout = 32s). Non-ssh connections (local sockets, tcp) keep the default (short) timeout, as those connections are either established quickly, or fail quickly when the daemon is not running, so their fast-fail behavior is unchanged.

- How I verified it

Reproduced the issue by simulating a slow ssh connection (an ssh wrapper that delays the connection by ~3 seconds, i.e. longer than defaultInitTimeout) against a local daemon with a lower API version than the client:

Before (client built from master):

$ PATH=/tmp/slowssh:$PATH DOCKER_HOST=ssh://localhost ./docker --debug version
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- localhost docker system dial-stdio]"
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- localhost docker system dial-stdio]"
 API version:       1.56

After:

$ PATH=/tmp/slowssh:$PATH DOCKER_HOST=ssh://localhost ./docker --debug version
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- localhost docker system dial-stdio]"
 API version:       1.55 (downgraded from 1.56)

The initial ping now completes within the longer timeout, the connection is reused for the subsequent request (only one ssh connection is started), and the API version is negotiated correctly.

Also added unit tests:

  • TestGetInitTimeout: verifies the timeout selection per endpoint type (default for unix/tcp, longer for ssh, explicit initTimeout override takes precedence for both).
  • TestInitializeFromClientSlowConnection: simulates a daemon that is slow to connect to (dial takes longer than defaultInitTimeout), and verifies that for an ssh:// endpoint the initial ping completes and the API version is negotiated, whereas non-ssh endpoints keep the default fast-fail behavior.

go test ./cli/command/... passes, and golangci-lint (v2.10.1, as used in CI) reports no issues.

The CLI performs an initial ping to the daemon during (lazy)
initialization, which is used for API-version negotiation and to
collect server info (e.g. whether experimental features are enabled,
and the daemon's OS type). This ping has a short (2 second) timeout
to prevent the CLI from hanging when connecting to a local daemon
that is not running.

Establishing a connection through the ssh connection helper requires
a TCP connection, host key verification, and authentication before
the connection to the daemon is ready to be used, which can take
longer than this timeout, in particular when connecting to a host
for the first time. In that case, the initial ping is aborted and
API-version negotiation is skipped: the client uses its default
(maximum) API version, and requests to a daemon with a lower API
version fail with errors such as:

    Error response from daemon: client version 1.56 is too new.
    Maximum supported API version is 1.55

This can be observed with 'docker --debug --context <ssh-context>
version', which dials the daemon twice on a slow (first) connection,
and prints the client's default (maximum) API version instead of the
negotiated version.

The ssh connection helper sets a default ConnectTimeout=30 for the
ssh process, so allow the initial ping for ssh connections to complete
within that time, plus some additional time for the handshake and the
ping itself. Non-ssh connections (local sockets, tcp) keep the default
(short) timeout, as those connections are either established quickly,
or fail quickly when the daemon is not running.

Fixes docker#6125

Signed-off-by: adisivaprasad <adisivaprasad@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

api-version negotiation sometimes fails over SSH connection

1 participant