Got an error with connecting daemon over ssh session

[frodo821]

With setting ssh URL to the DOCKER_HOST environment var, the docker CLI does not work.

I'm using an EC2 instance (amazon linux 2) to run the docker daemon and set the ssh URL as the DOCKER_HOST. Then I ran the docker containers command on my local computer, but it had been failed with the error: error during connect: Get http://docker/v1.40/containers/json: net/http: HTTP/1.x transport connection broken: malformed HTTP status code "pid"

Next, I ran the command on the instance over the ssh connection; it seems to be succeeded.

How to fix it? or is this a bug?

$ docker -v
Docker version 19.03.13, build 4484c46d9d

I'm using docker for mac, Mac OS Mojave (10.14.6).

Best Regards.

[ZirionNeft]

I,ve got the same issue. I guess is that is something wrong with target host, because I have another context, and it is working fine.

[mvoehringer]

I have almost the same issue connection to a EC2 running amazon linux. On some other hosts everything work fine.

error during connect: Get "http://docker.example.com/v1.24/containers/json": net/http: HTTP/1.x transport connection broken: malformed HTTP status code "Succeeded"

This is the versions on the EC2 container.

[ec2-user@#### ~]$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 21
  Running: 5
  Paused: 0
  Stopped: 16
 Images: 3
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: amazon-ecs-volume-plugin local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ####
 runc version: ####
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.252-195.483.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.689GiB
 Name: ####
 ID: D522:6LZ3:PYMX:NUEO:2IQG:6JEW:O5BA:BH3X:3M3W:DW5F:U5AE:R2T5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[zouhairm]

Hi - I'm running into the same issue, did any of you figure out a solution to this?

Thanks!

[zouhairm]

actually, small clarification. The status code my http error is showing is ":" (not pid or success as reported above)

ERROR: error during connect: Get "http://docker.example.com/v1.24/info": net/http: HTTP/1.x transport connection broken: malformed HTTP status code ":"

[zouhairm]

figured it out for future folks landing here:
the remote instance had an 'echo' in bashrc (used for debugging something unrelated), which was tripping docker for some reason (presumably it wasn't expecting that in whatever output stream it was parsing)
removing it fixed it and things are working as expected now'

[DarrenBishop]

I can see that docker is invoking a login-shell as the error for me is

...malformed HTTP status code "/Users/darren/.ssh/*.vars..."

And I know exactly what that is and where it comes from (i.e. my .bashrc)

Is there a way to tell docker to not use a login-shell?
Alternatively, does docker populate any kind of signa, (e.g. env-var) that I can check for in my .bashrc?

[thaJeztah]

The CLI-side of what's done is fairly minimal;

cli/cli/connhelper/connhelper.go

Lines 50 to 57 in 41183c2

	Dialer: func(ctx context.Context, network, addr string) (net.Conn, error) {
	args := []string{"docker"}
	if sp.Path != "" {
	args = append(args, "--host", "unix://"+sp.Path)
	}
	sshFlags = addSSHTimeout(sshFlags)
	args = append(args, "system", "dial-stdio")
	return commandconn.New(ctx, "ssh", append(sshFlags, sp.Args(args...)...)...)

docker sets minimal options, other than a timeout, and the command to run over the connection; If you set the --debug flag, you can see the command that the CLI executes to connect;

docker --debug version
DEBU[0000] commandconn: starting ssh with [-o ConnectTimeout=30 -- swarm-test-01 docker system dial-stdio]

Other than that, it depends on your SSH configuration, so if there's an option to disable such things there (I'm horrible at all options that can be set there, so don't have good recommendations for that), you could make changes to your SSH config.

[DarrenBishop]

@thaJeztah thanks for your comments

I managed to fix it, but got lucky in a sense; that "/Users/darren/.ssh/.vars..." is some output I emit from my ~/.bash_ init scripts.
I have loads of such output, but it was the only one that prints to stdout, while all others I have redirected to stderr (i.e. which I treat as a debug stream).

The thing is, I use Bash-It and load all sorts of completions, etc, into my Bash session, so it'll only take one of any such thing to 'debug to stdout' to cause this issue and won't have any control over it.

Bearing all that in mind, I don't see why docker needs to be run (remotely) with all the remote user's shell-init cruft loaded

[trajano]

It may be a bug but the connection via SSH should be equivalent to
ssh -T