[Docker Engine] Add multi-platform image support (migrate to the containerd content store and snapshotters)

Tell us about your request
Docker started the containerd project with the plan for the Docker Engine to be migrated to use it. The Docker Engine already uses containerd to manage container lifecycle but it still uses its own implementation for image and filesystem management.

Docker is working with the Moby project contributors on completing the migration and moving Docker Engine to use containerd's content store and snapshotters. This will enable low-level features like multi-platform image support. It will also reduce the code complexity of the Docker Engine.

Which service(s) is this request for?
Docker Engine which is a core part of Docker Desktop and Docker for Linux.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
There are many user facing features that this will enable. To list a few:

Native multi-platform image build support (i.e.: do not require containerized BuildKit)
Resource namespacing
More OCI support
Support for containerd’s snapshotters

The containerd image store feature is now in beta in Docker Desktop 4.12.0 🎉
More info is available here: https://docs.docker.com/desktop/containerd/

We’d love to hear from you! Please feel free to provide us feedback or report any bugs you may find through the issues tracker on the feedback form.

Building with --platform works, but gives this warning:

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

Using--load as suggested by the warning results in this error:

ERROR: docker exporter does not currently support exporting manifest lists

Can you explain what is supposed to happen here and why following the instructions does not work?

Which instructions were you following? https://docs.docker.com/desktop/containerd/#building-multi-platform-images ?

There's not really that much detail in your comment, I'd suggest opening a Docker Desktop bug report for the failure you're seeing, and hence (by populating the issue template) including details like the commands you ran, the Dockerfile, and the versions of the various components in-use.

I suspect that something in your configuration (the use of the buildx Docker container driver, perhaps) is incompatible with the containerd image store for the Docker component versions currently installed, and hence guess that you will need to manually upgrade to buildx 0.10 and/or wait for a new release of Docker Desktop to include buildx 0.10 and hence BuildKit 0.11, which I think fixes this particular incompatibility.

To be clear, that's several guesses stacked together, so I still recommend opening a new bug report for the problem with all the details on the likely chance I'm giving incorrect or faulty suggestions.

Which instructions were you following?

These instructions:

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

So, I append --load to the command, and then I get this message instead:

ERROR: docker exporter does not currently support exporting manifest lists

Okay, if you weren't following a specific tutorial (sorry, I misunderstood) then the answer is there in the output: If using the docker-container driver, you must provide either --push or --load in order to get the built manifest or manifest list out of the build cache (this is true irrespective of the containerd integration, and is as-expected), and the docker exporter (triggered by --load) does not handle manifest lists in that version of buildx (which is also not related to the containerd image store; the cause here is that without the containerd content store, docker doesn't know how to store manifest lists at all, so the docker exporter didn't support them either).

Basically, you've hit docker/buildx#59 (comment). I still guess that the fix is to enable both the containerd content store beta (as you are) so that manifest lists can be stored and also the newer version of buildx I mentioned earlier, which knows how to store manifest lists in Docker. (Note: I have not tested this, just looked at some PRs)

Does any version of buildx support multi-platform exports with the beta containerd store? My understanding is that exporting an image to docker (--load) still explicitly blocks exporting multi-platform images:

https://github.com/docker/buildx/blob/fb27e3f919dcbf614d7126b10c2bc2d0b1927eb6/build/build.go#L507-L509

Drat. That's why it was a guess. 🐍👀.

My guess was based on moby/buildkit#3389 having been merged for BuildKit 0.11, and BuildKit 0.11 having been vendored into buildx 0.10. However, on closer inspection, it (and moby/buildkit#3380 which is how I got there) are not solving this issue, but were preventing the same error message being triggered by other changes.

So I guess there's still some work to go in buildx to be able to push a manifest list via the Docker exporter (--load) into the content store when containerd is enabled, and perhaps even work needed in the Docker Engine side and/or BuildKit too.

I wasn't able to find an open issue tracking this: docker/buildx#59 was the first one, and it was closed by documenting the limitation with a direction towards this roadmap issue as the blocker. docker/buildx#166 is also close, but it is related to the image exporter used by the docker driver, and that's apparently working with this beta already.

Again, it's possible that a specific tracking issue exists, and I just failed to locate it.

If I'm reading this correctly, then BuildKit would also reject --load (shorthand for --output=type=docker) if the relevant check was removed from buildx. That block is only hit for the Docker on-disk image format, as it looks like the OCI on-disk image format does support manifest lists (see this block) and so the required work is make ~~this code~~ edit: mistake, it should be this branch work for an OCI format tarball (probably needs to be done in Docker Engine? Maybe already works?) and change this block a bit so that you don't need to do --output=type=oci,dest= (or whatever trick is needed to get an empty-string "dest" value into the CLI parser) to cause the previous-linked block to be hit in OCI mode.

And just to clarify, this is only affecting the docker-container buildx driver. The docker buildx driver uses the image exporter instead, which should already work for the equivalent of --load with the containerd beta storage as demonstrated at https://docs.docker.com/desktop/containerd/#building-multi-platform-images.

@TBBle

Excellent write-up @TBBle, thanks! So, just for the record, this is the current setup:

docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:41 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.16.1 (95567)
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:43 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker buildx version
github.com/docker/buildx v0.10.0 876462897612d36679153c3414f7689626251501

And the failing command is:

docker buildx build --platform=linux/amd64,linux/arm64 --no-cache --pull--load --tag ourregistry/ourproject:ourrevision

fix: only run bash tests against the non-buildx version

feat: rootfs - build/upload action and Dockerfile (#125)

I am compiling x86 on my M2 machine, and I enabled containerd on Docker.
This gives me error parent snapshot sha256:b6bab4b00958273f1ba360d4fbb20cf6b4ee829508f4b4985e16d51a28b3a0f0 does not exist: not found

@zhixinwen

@zhixinwen The containerd backend for Docker is still considered experimental, so I'm afraid that's likely an issue with Docker for Mac.

Any updates here?

What's the timeline until we will be able to use multi-platform builds using docker buildx build --platform=linux/amd64,linux/arm64 it's been a while since you wrote about it at https://www.docker.com/blog/multi-platform-docker-builds/

Containerd image store support is now generally available with the release of Moby 25 (shipped in DD 4.27). There will be a blog post that covers the DD 4.27 soon. Docs can be found at https://docs.docker.com/desktop/containerd/.

Looking forward to additional documentation because the link did not work with containerd and SBOM enabled:

❯ docker buildx create --bootstrap

[+] Building 5.5s (1/1) FINISHED                                                                                                                                                                                                                                                      
 => [internal] booting buildkit                                                                                                                                                                                                                                                  5.5s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                                                                                               4.9s
 => => creating container buildx_buildkit_pedantic_brattain0                                                                                                                                                                                                                     0.6s
pedantic_brattain
❯ docker info --format="{{ .Driver }}"

overlay2
❯ docker buildx build --platform=linux/arm64,linux/amd64 --tag=user/containerd-multiplatform ./docker/proxy-nginx
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                          docker:desktop-linux
ERROR: Multi-platform build is not supported for the docker driver.

@NiklasBr

hi @NiklasBr. From your docker info command it looks like you aren't using the containerd storage backend.

docker info --format="{{ .Driver }}"
overlay2

overlay2 is the default graph driver. If you enable the containerd image store feature in Docker Desktop and try again, building multi-platform images and loading them to the local image store should work fine, even with the default buildx driver.

Assuming there were more lines to that error

Switch to a different driver, or turn on the containerd image store, and try again.
Learn more at https://docs.docker.com/go/build-multi-platform/

then it appears your docker buildx doesn't believe you're using the containerd image store.

If no such lines appeared, then I have no idea what version of buildx you're using, but I can't see any way in the buildx 0.12.1 release to generate that error without those extra lines, and 0.11 and older had different phrasing for that error.

I just upgraded my Docker Desktop to 4.27.1 (I already had the containerd image store enabled from the beta), and trivially built the container from https://docs.docker.com/get-started/02_our_app/ using the given command:

git clone https://github.com/docker/getting-started-app.git
cd .\getting-started-app
# Populate the Dockerfile from the example
gvim .\Dockerfile
docker buildx build --platform=linux/arm64,linux/amd64 --tag=user/containerd-multiplatform .

and it appears to have worked as expected:

> docker image list
REPOSITORY                      TAG       IMAGE ID       CREATED              SIZE
user/containerd-multiplatform   latest    03d0489bce84   About a minute ago   86.6MB
user/containerd-multiplatform   latest    03d0489bce84   2 minutes ago        352MB

Navigation Menu

Repository navigation

[Docker Engine] Add multi-platform image support (migrate to the containerd content store and snapshotters) #371

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Docker Engine] Add multi-platform image support (migrate to the containerd content store and snapshotters) #371

Description

Activity

sam-thibault commented on Sep 14, 2022

NiklasBr commented on Jan 12, 2023

TBBle commented on Jan 12, 2023

NiklasBr commented on Jan 12, 2023

TBBle commented on Jan 12, 2023

jacobwgillespie commented on Jan 13, 2023

TBBle commented on Jan 13, 2023

NiklasBr commented on Jan 16, 2023

9 remaining items

zhixinwen commented on Sep 8, 2023

AaronFriel commented on Sep 10, 2023

NiklasBr commented on Dec 1, 2023

NiklasBr commented on Feb 8, 2024

savannahostrowski commented on Feb 8, 2024

NiklasBr commented on Feb 8, 2024

dvdksn commented on Feb 8, 2024

TBBle commented on Feb 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions