Repository navigation
Conversation
…erence A pull moves an image reference and leaves every running container on the image it was created from. Nothing said so, so long-lived workspaces drifted days behind (#673). The check reads devpod's workspace_result.json for the container id, the image it was created from and the image the devcontainer declared. It asks docker for the container's image and the reference's current image. Where devpod derived an image on top of the declared one, the container is stale when the declared image's layers are no longer the start of the running image's layers. Every doubt reads as current. `dl --ls` prints one line per stale workspace under the table. `dl --ls --json` appends `staleImage` to the row, absent otherwise, so a machine with nothing stale prints the same document as before. Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…own container A reference whose layers all appear in the running image but not as its prefix now reads as stale, and two workspaces behind one reference are each judged by their own container, with the reference inspected once.
The clean row now carries a stale image before the mark, so a mark that skips rows absent from the answer leaves it standing and the test fails.
The layer rule said "different", not "older". A prebuilt derived image pulled from a registry can sit on a newer base than this machine's copy of the declared reference. Its layers then differ, the note said stale, and a recreate pulled the same prebuild again, so the note never cleared. The derived case now also asks that the declared image was created after the running one. docker's Created is cut to whole seconds so two of them compare in time order as strings. Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
Reviewer's GuideThis PR detects workspaces whose running containers lag behind their current image references by combining devpod's workspace_result.json data with bounded, batched Docker inspections, then surfaces actionable recreate guidance on stderr and an optional staleImage field in --ls JSON while preserving the existing table and clean-output behavior. Sequence diagram for stale workspace detection during dl --lssequenceDiagram
participant CLI as dl --ls
participant Listing as Listing flow
participant Devpod as workspace_result.json
participant Docker as Docker
CLI->>Listing: stale_images(runner, devpod_home, workspace_ids)
Listing->>Devpod: Read ContainerDetails and MergedConfig.image
Listing->>Docker: inspect_formatted(container, container ids)
Docker-->>Listing: Running image ids
Listing->>Docker: inspect_formatted(image, running image ids)
Docker-->>Listing: Running image metadata
loop Each distinct image reference
Listing->>Docker: inspect_formatted(image, reference)
Docker-->>Listing: Current image metadata
end
Listing-->>CLI: StaleImages
alt Table output
CLI->>CLI: stale_image_notes(workspace, reference)
CLI-->>CLI: Write recreate guidance to stderr
else JSON output
CLI->>Listing: mark_stale_images(rows, stale)
CLI-->>CLI: Append staleImage only on stale rows
end
Flow diagram for stale image classificationflowchart TD
A[Read workspace_result.json] --> B{Container and reference available?}
B -- No --> C[Treat as current]
B -- Yes --> D[Inspect running container image]
D --> E{Docker read succeeded?}
E -- No --> C
E -- Yes --> F[Inspect current reference image]
F --> G{Reference read succeeded?}
G -- No --> C
G -- Yes --> H{Running image id equals current id?}
H -- Yes --> C
H -- No --> I{Container created from reference?}
I -- Yes --> J[Mark stale]
I -- No --> K{Reference layers are not running-image prefix and reference is newer?}
K -- Yes --> J
K -- No --> C
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 7 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="rust/devlaunch-core/src/flows/mod.rs" line_range="46" />
<code_context>
// it is one decision and no socket.
pub mod session_manager;
// binary surface — not part of the frozen wf API (#251 §7)
+pub mod stale_images;
+// binary surface — not part of the frozen wf API (#251 §7)
pub mod workspace_clone;
</code_context>
<issue_to_address>
**Public API checks fail**
When the public-API CI job runs against this change, the new public `stale_images` API adds generated items that are absent from the checked-in snapshots, so the job fails and blocks the PR.
Regenerate and commit the updated public-API snapshots.
Also at `rust/devlaunch-core/src/flows/stale_images.rs:146`, `rust/devlaunch-core/src/flows/listing.rs:1013`.
</issue_to_address>
### Comment 2
<location path="rust/devlaunch-core/src/flows/stale_images.rs" line_range="195" />
<code_context>
+ if created.created_from == created.reference {
+ return true;
+ }
+ !running.layers.starts_with(¤t.layers) && current.created > running.created
+}
+
</code_context>
<issue_to_address>
**Changed bases go undetected**
When a derived image’s declared base has changed but its `Created` timestamp is equal to or earlier than the derived image’s timestamp, or the timestamps come from skewed clocks, `is_stale` relies on `current.created > running.created` to detect a changed base, so it misses stale workspaces and can also recommend recreating a workspace when that will not clear the warning.
Determine whether the declared base changed without relying on creation-time ordering.
Also at `rust/devlaunch-core/src/flows/stale_images.rs:262`.
</issue_to_address>
### Comment 3
<location path="rust/devlaunch-core/src/flows/stale_images.rs" line_range="192-193" />
<code_context>
+ if running.id == current.id {
+ return false;
+ }
+ if created.created_from == created.reference {
+ return true;
+ }
+ !running.layers.starts_with(¤t.layers) && current.created > running.created
</code_context>
<issue_to_address>
**Rollbacks trigger false warnings**
When a direct image reference is retagged to an image older than the image the container runs, `is_stale` returns `true` for every differing image ID when `created_from == reference`, so `dl --ls` recommends recreating the workspace onto the older image.
Check whether the referenced image is newer before marking a direct-reference workspace stale.
</issue_to_address>
### Comment 4
<location path="rust/devlaunch-core/src/flows/stale_images.rs" line_range="262" />
<code_context>
+ id.to_owned(),
+ Image {
+ id: id.to_owned(),
+ created: created.get(..19)?.to_owned(),
+ layers,
+ },
</code_context>
<issue_to_address>
**Same-second builds look current**
When a derived image and its newer declared reference have different layers but are created within the same second, `images` truncates creation timestamps to whole seconds, so `is_stale` sees equal times and omits the stale workspace.
Preserve sufficient timestamp precision when parsing Docker creation times.
Also at `rust/devlaunch-core/src/flows/stale_images.rs:195`, `rust/devlaunch-core/src/flows/stale_images.rs:263`.
</issue_to_address>
### Comment 5
<location path="rust/devlaunch-core/src/flows/stale_images.rs" line_range="201" />
<code_context>
+/// What devpod's result file says about this workspace's container, if there is
+/// exactly one result file and it names a container and an image.
+fn created(devpod_home: Option<&DevpodHome>, workspace_id: &str) -> Option<Created> {
+ let path = sole_workspace_result(devpod_home, workspace_id)?;
+ let document: serde_json::Value = serde_json::from_slice(&std::fs::read(path).ok()?).ok()?;
+ let details = &document["ContainerDetails"];
</code_context>
<issue_to_address>
**Duplicate IDs appear current**
When the listing contains the same workspace ID in multiple devpod contexts, `created` looks up results by workspace ID alone, and `sole_workspace_result` returns no result for an ID found in multiple contexts. The workspaces are treated as current and receive no stale warning or `staleImage` field.
Look up each workspace result using its context as well as its ID.
Also at `rust/dl/src/commands.rs:348-349`, `rust/dl/src/commands.rs:414`, `rust/devlaunch-core/src/flows/stale_images.rs:146-153`.
</issue_to_address>
### Comment 6
<location path="rust/devlaunch-core/src/flows/stale_images.rs" line_range="207" />
<code_context>
+ let created_from = text(&details["Config"]["Image"])?;
+ Some(Created {
+ container: text(&details["Id"])?,
+ reference: text(&document["MergedConfig"]["image"]).unwrap_or_else(|| created_from.clone()),
+ created_from,
+ })
</code_context>
<issue_to_address>
**Compose workspaces hide stale bases**
When A compose workspace has no `MergedConfig.image` and its service image is pulled to a newer version, `created` falls back to `created_from` when `MergedConfig.image` is absent, so a compose workspace’s devpod-generated image tag becomes both the created image and the reference. Pulling a newer image for the compose service does not move that tag, and `dl --ls` reports no stale workspace.
Resolve the compose service’s declared image reference for the workspace, or explicitly represent the check as unsupported instead of treating the generated tag as the service reference.
</issue_to_address>
### Comment 7
<location path="rust/devlaunch-core/src/flows/stale_images.rs" line_range="162-166" />
<code_context>
+ let running_images = images(runner, unique(running.values()));
+ let references: BTreeMap<String, Image> = unique(created.iter().map(|(_, c)| &c.reference))
+ .into_iter()
+ .filter_map(|reference| {
+ let image = images(runner, [reference.clone()]).into_values().next()?;
+ Some((reference, image))
+ })
+ .collect();
+
+ let by_workspace = created
</code_context>
<issue_to_address>
**Many references delay listings**
When many listed workspaces have distinct references and Docker is slow or unresponsive, the reference loop calls `images` sequentially, and each inspection has a 10-second timeout, so `dl --ls` can wait up to 10 seconds per reference before returning the table and notes.
Inspect the distinct references in a batched or parallel operation.
</issue_to_address>| if created.created_from == created.reference { | ||
| return true; | ||
| } | ||
| !running.layers.starts_with(¤t.layers) && current.created > running.created |
There was a problem hiding this comment.
🟠 High · Changed bases go undetected
When a derived image’s declared base has changed but its Created timestamp is equal to or earlier than the derived image’s timestamp, or the timestamps come from skewed clocks, is_stale relies on current.created > running.created to detect a changed base, so it misses stale workspaces and can also recommend recreating a workspace when that will not clear the warning.
Determine whether the declared base changed without relying on creation-time ordering.
Also at rust/devlaunch-core/src/flows/stale_images.rs:262.
Prompt for AI agents
In `rust/devlaunch-core/src/flows/stale_images.rs` at line 195:
**Changed bases go undetected**
When a derived image’s declared base has changed but its `Created` timestamp is equal to or earlier than the derived image’s timestamp, or the timestamps come from skewed clocks, `is_stale` relies on `current.created > running.created` to detect a changed base, so it misses stale workspaces and can also recommend recreating a workspace when that will not clear the warning.
Determine whether the declared base changed without relying on creation-time ordering.
Also at `rust/devlaunch-core/src/flows/stale_images.rs:262`.
Comment on lines
+192
to
+193
| if created.created_from == created.reference { | ||
| return true; |
There was a problem hiding this comment.
🟡 Medium · Rollbacks trigger false warnings
When a direct image reference is retagged to an image older than the image the container runs, is_stale returns true for every differing image ID when created_from == reference, so dl --ls recommends recreating the workspace onto the older image.
Check whether the referenced image is newer before marking a direct-reference workspace stale.
Prompt for AI agents
In `rust/devlaunch-core/src/flows/stale_images.rs` at lines 192-193:
**Rollbacks trigger false warnings**
When a direct image reference is retagged to an image older than the image the container runs, `is_stale` returns `true` for every differing image ID when `created_from == reference`, so `dl --ls` recommends recreating the workspace onto the older image.
Check whether the referenced image is newer before marking a direct-reference workspace stale.| id.to_owned(), | ||
| Image { | ||
| id: id.to_owned(), | ||
| created: created.get(..19)?.to_owned(), |
There was a problem hiding this comment.
🟡 Medium · Same-second builds look current
When a derived image and its newer declared reference have different layers but are created within the same second, images truncates creation timestamps to whole seconds, so is_stale sees equal times and omits the stale workspace.
Preserve sufficient timestamp precision when parsing Docker creation times.
Also at rust/devlaunch-core/src/flows/stale_images.rs:195, rust/devlaunch-core/src/flows/stale_images.rs:263.
Prompt for AI agents
In `rust/devlaunch-core/src/flows/stale_images.rs` at line 262:
**Same-second builds look current**
When a derived image and its newer declared reference have different layers but are created within the same second, `images` truncates creation timestamps to whole seconds, so `is_stale` sees equal times and omits the stale workspace.
Preserve sufficient timestamp precision when parsing Docker creation times.
Also at `rust/devlaunch-core/src/flows/stale_images.rs:195`, `rust/devlaunch-core/src/flows/stale_images.rs:263`.| /// What devpod's result file says about this workspace's container, if there is | ||
| /// exactly one result file and it names a container and an image. | ||
| fn created(devpod_home: Option<&DevpodHome>, workspace_id: &str) -> Option<Created> { | ||
| let path = sole_workspace_result(devpod_home, workspace_id)?; |
There was a problem hiding this comment.
🟡 Medium · Duplicate IDs appear current
When the listing contains the same workspace ID in multiple devpod contexts, created looks up results by workspace ID alone, and sole_workspace_result returns no result for an ID found in multiple contexts. The workspaces are treated as current and receive no stale warning or staleImage field.
Look up each workspace result using its context as well as its ID.
Also at rust/dl/src/commands.rs:348-349, rust/dl/src/commands.rs:414, rust/devlaunch-core/src/flows/stale_images.rs:146-153.
Prompt for AI agents
In `rust/devlaunch-core/src/flows/stale_images.rs` at line 201:
**Duplicate IDs appear current**
When the listing contains the same workspace ID in multiple devpod contexts, `created` looks up results by workspace ID alone, and `sole_workspace_result` returns no result for an ID found in multiple contexts. The workspaces are treated as current and receive no stale warning or `staleImage` field.
Look up each workspace result using its context as well as its ID.
Also at `rust/dl/src/commands.rs:348-349`, `rust/dl/src/commands.rs:414`, `rust/devlaunch-core/src/flows/stale_images.rs:146-153`.| let created_from = text(&details["Config"]["Image"])?; | ||
| Some(Created { | ||
| container: text(&details["Id"])?, | ||
| reference: text(&document["MergedConfig"]["image"]).unwrap_or_else(|| created_from.clone()), |
There was a problem hiding this comment.
🟡 Medium · Compose workspaces hide stale bases
When A compose workspace has no MergedConfig.image and its service image is pulled to a newer version, created falls back to created_from when MergedConfig.image is absent, so a compose workspace’s devpod-generated image tag becomes both the created image and the reference. Pulling a newer image for the compose service does not move that tag, and dl --ls reports no stale workspace.
Resolve the compose service’s declared image reference for the workspace, or explicitly represent the check as unsupported instead of treating the generated tag as the service reference.
Prompt for AI agents
In `rust/devlaunch-core/src/flows/stale_images.rs` at line 207:
**Compose workspaces hide stale bases**
When A compose workspace has no `MergedConfig.image` and its service image is pulled to a newer version, `created` falls back to `created_from` when `MergedConfig.image` is absent, so a compose workspace’s devpod-generated image tag becomes both the created image and the reference. Pulling a newer image for the compose service does not move that tag, and `dl --ls` reports no stale workspace.
Resolve the compose service’s declared image reference for the workspace, or explicitly represent the check as unsupported instead of treating the generated tag as the service reference.
Comment on lines
+162
to
+166
| .filter_map(|reference| { | ||
| let image = images(runner, [reference.clone()]).into_values().next()?; | ||
| Some((reference, image)) | ||
| }) | ||
| .collect(); |
There was a problem hiding this comment.
🟡 Medium · Many references delay listings
When many listed workspaces have distinct references and Docker is slow or unresponsive, the reference loop calls images sequentially, and each inspection has a 10-second timeout, so dl --ls can wait up to 10 seconds per reference before returning the table and notes.
Inspect the distinct references in a batched or parallel operation.
Prompt for AI agents
In `rust/devlaunch-core/src/flows/stale_images.rs` at lines 162-166:
**Many references delay listings**
When many listed workspaces have distinct references and Docker is slow or unresponsive, the reference loop calls `images` sequentially, and each inspection has a 10-second timeout, so `dl --ls` can wait up to 10 seconds per reference before returning the table and notes.
Inspect the distinct references in a batched or parallel operation.Copied from the public-api CI job's diff. Only public-api.rest.txt moves; nothing in the promised api.txt changed. Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
This was referenced Oct 7, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #673 — A workspace keeps its old image after a pull, and a recreate ends its agent. This PR is part 1 of 3: it finds the stale workspaces. Part 2 makes
dl <ws> recreatekeep the Claude session. Part 3 addsdl --refresh-stale.What this changes
flows/stale_images.rs, finds each workspace whose container runs an older image than its image reference now names. This PR calls such a workspace stale.dl --lsprints one line on stderr under the table for each stale workspace. The table does not change:dl --ls --jsonadds astaleImagekey at the end of a stale row. The value is the image reference. The key is absent on every other row, so a machine with no stale workspace prints the same document as before.docker::inspect_formattedrunsdocker inspect --type <kind> --format <format> <names...>. It keeps stdout when docker exits 1, because docker prints a line for each name that it found and exits 1 if one name is missing.docs/agents-using-dl.mdandCHANGELOG.mddescribe the new key and the new note.Where this fits
devpod writes
workspace_result.jsonafter each completedup. The check reads three fields from it and then asks docker:The time check stops a false alarm. A prebuilt derived image from a registry can sit on a newer base than this machine's copy of the reference. Without the time check, the note would stay after each recreate.
If any read fails, the workspace reads as current. The reasons are: no result file, a container that docker no longer has, a reference that this machine never pulled, or no docker.
How I checked it
Before: no fixture wrote
Config.Image, so the--lstests never ran the check. After:dl/tests/read_side.rsputs a fakedockeronPATHand runs the real binary.dl --lsprints the note on stderr, and the table is the same. The--ls --jsonrow ends withstaleImage.stale_images.rshas 12 unit tests onFakeRunner. They cover the direct case, the derived case, the prebuild on a newer base, a missing container, a missing reference, no docker, and two workspaces that share one reference (one inspect of the reference).a_prebuilt_image_on_a_newer_base_than_the_local_reference_is_currentfailed.Commands that I ran in the devcontainer:
cargo test --workspace: all pass.cargo clippy --locked --all-targets -- -D warningsandcargo fmt --check: clean.pytest test/against the debug build: 845 passed.prek run --from-ref origin/main --to-ref HEAD: passed.I did not run:
scripts/public-api-snapshots.sh.cargo-public-apiand the nightly toolchain are not in the container. I copied the diff from thepublic-apiCI job intopublic-api.rest.txt.Merge danger
Door: two-way. The JSON key is additive and absent by default. A revert removes the note and the key.
Blast radius: small. If the rule is wrong,
dl --lsprints a wrong note, or misses one.dl --lsnow also makes 2 + Rdocker inspectcalls (R is the count of distinct references), each with a 10s limit. A docker daemon that hangs makesdl --lsslower.Review notes
MergedConfig.imageis empty. If devpod built a derived tag for the service, that tag does not move on a pull, so the check sees no change. The issue measured kinisi_ros. Someone must read one real kinisiworkspace_result.jsonon the host to confirm which case it is.ENVorLABEL) keeps every layer. A derived container then reads as current. This is a miss, not a false alarm.Stack
🤖 Generated with Claude Code