Does devpod record the image it built for a workspace #447

Activity

1. blooop

   added

   wayfinder:research

   on Aug 25, 2026
2. blooop

   added a parent issue on Aug 25, 2026

   • What dl reclaims and what it must leave #444
3. blooop

   marked this as blocking on Aug 25, 2026

   • Docker images and build cache: what does dl reclaim #450
4. blooop

   self-assigned this

   on Aug 25, 2026
5. blooop commented on Aug 25, 2026

   

   blooop

   on Aug 25, 2026

   OwnerAuthor

   More actions

   agent-decided (principle 2, no straggler data): the record exists.

   Findings: docs/research/devpod-image-record.md on research/devpod-image-record. Knowledge only. The policy is #450's, and nothing below picks it.

   devpod records the image reference per workspace, in one field of a file dl already parses. workspace_result.json → .ContainerDetails.Config.Image. That is the same document flows::lifecycle::devcontainer_volumes opens for SubstitutionContext.LocalWorkspaceFolder and SubstitutionContext.DevContainerID, through the same clients::devpod_home::sole_workspace_result contexts walk. So the volume precedent transfers exactly: the name is read from devpod's own record, not guessed from a pattern, and reading one more key off an already-parsed document is not an architecture change. workspace.json carries no image reference at all.

   Measured 2026-08-25 in this repo's devcontainer, docker 29.7.2-2, thirteen throwaway workspaces on a scratch DEVPOD_HOME, pixi run devpod (v0.26.1) plus 0.8.11 and 0.16.6 out of pixi exec. Not inferred from devpod's docs.

   The field is uniform across how the image got there. {"image": "alpine:3.20"} records alpine:3.20; a Dockerfile build records bld-c745b:devpod-9f56be24be1c0b2903995ed4cbfe8a9e, i.e. <folder>-<5 hex>:devpod-<32 hex>; a prebuildRepository hit records the pulled reference localhost:5000/dl447pre:devpod-a647918106520274321fd35ad936b5a4, with the same 32-hex tag the publishing devpod build used. And the record separates the cases without pattern-matching the tag: a non-null MergedConfig.image means devpod pulled a reference the devcontainer named (alpine:3.20, not devlaunch's to touch), null with a config.build beside it means the reference is one devpod derived.

   Stable across every version dl has ever pinned. 0.8.11, 0.16.6 and 0.26.1 all write the same four top-level keys, the same ContainerDetails.Config shape and the same tag form. The floor is devpod >=0.26.1 in pyproject.toml and conda.recipe/recipe.yaml, so the field predates it by the whole supported range. Incidental but load-bearing for anyone reading source: the pinned devpod is not built from loft-sh/devpod, whose newest release is v0.6.15; strings over the binary gives its module path as github.com/skevetter/devpod.

   Four places the record is not, all measured. These are facts, not arguments.

   • A create that dies in its lifecycle hooks leaves workspace.json and no result, while the image it built stands (fail-4e13f:devpod-9fddd6eb…). Same on 0.8.11. Identical shape to CreateRecord::NeverCompleted.
   • devpod delete removes the container and the whole workspace directory and leaves the image, so the only record naming it is destroyed at the moment it stops being needed. Anything acting on the reference reads it before the delete, which is the ordering flows::lifecycle already uses.
   • A rebuild after a content change leaves the old tagged image with nothing naming it. Tagged, so docker image prune without -a does not touch it. That is the mechanism behind the 86.5 GB.
   • devpod build, which is what pixi run devcontainer-prebuild runs, creates and deletes its own temporary workspace, leaving a local image no record ever named.

   Of the 14 :devpod-tagged images the session produced, 10 were named by a surviving record and 4 were not. That ratio is an artifact of a session that provoked every failure mode; what it fixes is the shape, that a read of the records is a lower bound and never the whole set.

   A label at build time: one route, and it moves the prebuild tag. devpod up 0.26.1 exposes no build options at all, no --label, --build-arg, --tag or --repository; the image-adjacent flags are --devcontainer-image (replaces the image, skipping the build), --fallback-image and --prebuild-repository. --extra-devcontainer-path, the one flag that looks like a per-launch injection point, merges runtime config and drops the build half: an override carrying both containerEnv and build.options, on a first up of a fresh folder, applied the env (MergedConfig.containerEnv.DL_OVERRIDE = "yes") and produced Config.Labels: null with an unchanged tag. What does work is build.options in the repository's own devcontainer.json, which reaches the docker build and lands on the image. It is also an input to the tag hash: the same folder read devpod-605e9efe… with the label option and devpod-fd10a682… without it. Since the prebuild lookup is for that exact tag, adding one to .devcontainer/devcontainer.json would miss ghcr.io/blooop/devlaunch-devcontainer until the prebuild workflow republished under the new hash, and a label whose value varies per workspace or per launch could never match a published prebuild at all. Absent build.options, every image devpod builds carries Config.Labels: null; the three labels devpod does write (dev.containers.id, devcontainer.metadata, devpod.user) go on the docker run line and live on the container only, so docs/cleanup.md's claim is right as measured.

   Sharing, and it is three different cases. Two workspaces over one folder recorded the identical reference, and after one was rebuilt the other's record still named the old tag while its container still ran on it, so one tag was "superseded" and "in use" at once. Two folders with byte-identical Dockerfiles produced two tags and two image ids whose RootFS.Layers are the same digests, so ids can share nearly all their bytes and a per-image byte count is not additive. And an unforced docker rmi refuses while any container refers to the image, running or stopped, so the only images it can take at all are those with no container.

   Cost of being wrong, for the record: a rebuild, not data. Deleting the image made the next up rebuild under the same tag, because the tag doubles as devpod's build cache key. One wrinkle worth carrying into #450: the rebuild resolved the base afresh and produced a different image id, so removal is recoverable but not byte-identical. Same "the hash covers the recipe, not what the recipe pulls" argument CLAUDE.md makes about the prebuild tag, from the other end.
6. blooop

   closed this as completedon Aug 25, 2026
7. blooop commented on Aug 25, 2026

   

   blooop

   on Aug 25, 2026

   OwnerAuthor

   More actions

   Source corroboration, and two caveats the running binary could not supply. A parallel read of devpod's own Go source agrees with every measurement above; the findings doc now carries the citations in a new section 6. Nothing here changes the answer, so the ticket stays closed. Three items are worth surfacing for #450.

   The field is structurally a tag, never an image id. ContainerDetailsConfig (pkg/devcontainer/config/container_details.go:23-34) is the unmarshal target for docker inspect (pkg/docker/helper.go:324-334), and it has no member for docker's top-level .Image digest. The container behind the record quoted above was running sha256:a44bf1cf424e… and the record kept only the tag. So the reference has to be resolved through docker to become an id, and the tag can have moved in between. Also: devpod's own non-test code never reads the field back. The only reader is its e2e suite (e2e/tests/up/dockerfile_build.go:86,103,132,150), which uses it to detect whether a rebuild changed the image.

   The file is undocumented. workspace_result.json appears nowhere in devpod's docs/ tree. Its shape has in fact been unchanged since loft-sh v0.5.0, and the JSON key Image under ContainerDetails.Config goes back to loft-sh v0.1.0, but that is a track record and not a published contract. clients::devpod_home is already the module that absorbs exactly this risk for the volume names.

   Only the docker driver was established. The kubernetes driver builds its ContainerDetails by hand (pkg/driver/kubernetes/find.go:49-58) and the custom driver unmarshals whatever a provider prints (pkg/driver/custom/custom.go:67). Whether either populates Config.Image was not traced. Moot for dl as it stands, and it would stop being moot if that changed.

   The rest lines up exactly. The single write is provider2.SaveWorkspaceResult(...) at cmd/up.go:452, at the tail of devPodUp after every failing branch has already returned, which is the code behind "a failed create has no result". GetImageName (pkg/devcontainer/build/options.go:221-226) is basename + "-" + hash(folder)[:5] + ":" + prebuildHash, and CalculatePrebuildHash (pkg/devcontainer/config/prebuild.go:33-72) hashes architecture, a normalized config, the Dockerfile content and a .dockerignore-aware context hash, with ConfigBuildOptions.Options inside the normalized config, which is why build.options moves the tag. The prebuild lookup is that same string (pkg/devcontainer/build.go:341). The buildx command builder (pkg/driver/docker/build.go:95-105) never emits --label and never passes BuildOptions.Labels, which is why the built image has none, and that is inherited from loft-sh rather than a fork behaviour. --extra-devcontainer-path cannot reach a build for two independent reasons: AddConfigToImageMetadata (pkg/devcontainer/config/metadata.go:18-30) copies into an ImageMetadata with no DockerfileContainer and no ImageContainer, and its call site (pkg/devcontainer/build.go:46-55) runs after the build returned. And the delete path runs plain docker rm with no -v (pkg/docker/helper.go:150-157); a grep of pkg/, cmd/ and providers/ for rmi or image pruning finds nothing, so devpod never removes an image anywhere.

   One correction to the coverage figure in the comment above: nine of thirteen images were named by a surviving record, not ten of fourteen. proj-d3c2f:devpod-099e… on that daemon came from the source-reading run, not from these workspaces.