Repository navigation
[BUG] docker compose build produces different image IDs on each rebuild with containerd image store, but works correctly with overlay2 #13636
Description
Description
When Docker Engine uses the containerd image store (default for new installations since
Docker 28), docker compose build produces a new image ID on every rebuild even when no
files have changed. This causes docker compose up to unnecessarily recreate all containers
on every deploy.
Disabling the containerd image store on the same server fixes the issue — the second build
reuses the cache completely and the image ID stays the same.
Steps To Reproduce
Use Docker CE on Linux with the containerd image store enabled (default for fresh installs):
$ docker info -f '{{.Driver}}'
overlayfs
Create these three files:
Dockerfile:
FROM alpine:3.21
RUN echo "hello" > /hello.txt
COPY config.txt /config.txt
CMD ["cat", "/hello.txt", "/config.txt"]config.txt:
static config file
docker-compose.yml:
services:
app:
build: .
container_name: test-cacheThen run:
# Build twice with nothing changed in between:
docker compose build
docker images --format '{{.ID}}' --filter reference='*-app'
docker compose build
docker images --format '{{.ID}}' --filter reference='*-app'Expected behavior
Second build reuses cache. Image ID is the same. docker compose up does not recreate the
container.
Actual behavior
Second build produces a different image ID. docker compose up sees a different image and
recreates the container.
Note: individual build steps show CACHED for RUN/COPY layers, but the final image still
gets a new ID. The exporting steps (exporting layers, writing image, naming) take
0.1–0.2s instead of 0.0s, suggesting they actually re-export rather than reuse.
Compose Version
Docker Compose version v5.1.0
Docker Environment
Client: Docker Engine - Community
Version: 29.3.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.31.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v5.1.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 8
Running: 7
Paused: 0
Stopped: 1
Images: 8
Server Version: 29.3.0
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: local
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 301b2dac98f15c27117da5c8af12118a041a31d9
runc version: v1.3.4-0-gd6d73eb8
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-106-generic
Operating System: Ubuntu 24.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.57GiB
Name: secondary
ID: 091526cd-489b-4f47-acad-33efcea821cb
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Firewall Backend: iptables
Anything else?
Related issues
docker compose builddoes not completely reuse cached layers on immediate rebuild, butdocker builddoes for-mac#7341 — same symptom on macOS Docker Desktop- Layer not cached under
containerdimage store for-mac#7359 — layer not cached under containerd image store (fixed by increasing
defaultKeepStorage, but that does not help here — layers are small)
Workaround
Disable the containerd image store:
{
"features": {
"containerd-snapshotter": false
}
}After changing daemon.json and restarting Docker, existing images are no longer available
(format is incompatible) and need to be rebuilt/re-pulled. Volumes are preserved.
Activity
ndeloof commentedon Mar 19, 2026
Can be reproduce with bake:
Sounds like a Buildx Bake issue - cc @crazy-max
crazy-max commentedon Mar 20, 2026
The stronger hypothesis is Buildx/BuildKit export behavior when the containerd image store is enabled.
With the containerd image store, local images can retain attestations/index data. Buildx enables minimal provenance attestations by default. Provenance metadata includes per-build fields like invocation/timestamps, so you can get a different final image/index digest even when all filesystem layers are cached. That matches the observed symptom: build steps are
CACHED, but the final local image ID still changes.So containerd image store is likely exposing the problem, not causing it. The overlay2 "works" here mostly because the old image store does not preserve the same metadata/attestation shape locally.
A useful confirmation would be to rerun the repro with provenance disabled, for example:
docker compose build --provenance=false ...BUILDX_NO_DEFAULT_ATTESTATIONS=1 docker compose buildSee also related issues:
powerman commentedon Mar 20, 2026
I confirm both fixes the issue.
What's the point in updating timestamps when nothing else changes?
I mean, isn't this a bug anyway? Disabling this feature just to make caching work sounds weird.
crazy-max commentedon Mar 20, 2026
If this Buildx/BuildKit behavior is the expected shape of attested images, then the part that looks wrong here is probably Compose's equality check rather than the build result itself.
This is also how registry builds work. Once attestations are attached, the published artifact is the top-level image or index descriptor, and its digest legitimately changes when provenance metadata changes, even if the runnable image content does not.
The important detail is that Compose should not compare that top-level attested image/index digest for this use case. That digest reflects the whole published artifact, including attestations, not just the runnable image.
If Compose wants to answer "did the built image actually change in a way that should recreate containers?", it probably needs to compare the subject platform manifest digest, or possibly the image config digest, instead of the top-level attested index digest.
Otherwise provenance-only metadata churn will continue to look like an image change and will trigger unnecessary container recreation.
So my current understanding is that this likely needs to be handled on the Compose side by using a better equality boundary for rebuilt images.
RozbehSharahi commentedon Jun 11, 2026
I can confirm this. The mentioned workaround did work for us.
In our case nginx reverse-proxy along with all other containers would constantly recreate on
docker compose up -d --build. Once applied the workaround todaemon.jsonthis stopped and only containers which actually had changes would be recreated.fix(build): use platform image-manifest digest, not attested index
ci(e2e): run e2e against the containerd image store
fix(images): tolerate containers whose image record is gone
fix(images): tolerate containers whose image record is gone
docs: record docker image hygiene conventions and isolated-run cleanup (
fix(provider): address review findings on image distribution
fix(provider): address review findings on image distribution