Repository navigation
[Bug]: pullImage error when the image is actually pulled successfully #2503
Description
Steps to reproduce
Run a job on a VM-based backend that will require pulling a Docker image. For example, use a custom Docker image that is not yet pulled on the instance.
The issue happens randomly, you may need to try several times to reproduce.
Actual behaviour
The job fails:
NAME BACKEND INSTANCE RESOURCES RESERVATION PRICE STATUS SUBMITTED ERROR
httpbin datacrunch (FIN-01) CPU.4V.16G 4xCPU, 16GB, 100.0GB (disk), SPOT $0.01 failed 10:15 JOB_FAILED
(CREATING_CONTAINER_ERROR)
httpbin provisioning completed (failed)
Run failed with error code CREATING_CONTAINER_ERROR.
Error: pullImage error: failed pulling kennethreitz/httpbin:latest: downloaded 211188154/214167220 bytes (21.26MB/s), errors: []
Check CLI, server, and run logs for more details.
shim.log:
time=2025-04-11T08:15:30.303305Z level=debug msg=Pulling image
time=2025-04-11T08:15:39.773158Z level=debug msg=Status: Downloaded newer image for kennethreitz/httpbin:latest
time=2025-04-11T08:15:39.779208Z level=error msg=pullImage error: failed pulling kennethreitz/httpbin:latest: downloaded 211188154/214167220 bytes (21.26MB/s), errors: []
time=2025-04-11T08:15:39.780045Z level=debug msg=released GPU(s) gpus=[] task=f14ca241-2bfa-4c0d-af09-10b2af0d3912
time=2025-04-11T08:15:39.780085Z level=error msg=failed to run task=f14ca241-2bfa-4c0d-af09-10b2af0d3912 err=failed pulling kennethreitz/httpbin:latest: downloaded 211188154/214167220 bytes (21.26MB/s), errors: []
The image is actually pulled successfully to the runner instance
# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
kennethreitz/httpbin latest b138b9264903 6 years ago 534MBExpected behaviour
If the image is pulled successfully, shim does not fail with pullImage error and proceeds to start the runner.
dstack version
0.19.2
Server logs
Additional information
No response
Activity
r4victor commentedon Apr 11, 2025
Is it because of
downloaded 211188154/214167220 bytes? Any idea why the number of bytes does not match?jvstme commentedon Apr 11, 2025
Yes.
No, I haven't investigated yet.
ASmedberg-woolpert commentedon Apr 24, 2025
I'm hitting this too. Looks like the same error:
{"timestamp": "2025-04-24 18:00:35,903", "logger": "dstack._internal.server.background.tasks.process_running_jobs", "level": "WARNING", "message": "shim failed to execute job p1-mn10-0-0: creating_container_error (pullImage error: failed pulling ********.dkr.ecr.us-east-2.amazonaws.com/********: downloaded 5266659108/5268734644 bytes (48.24MB/s), errors: [])"}I'm not super familiar with the intricacies of docker or dstack, so not sure what other information I could provide that would help. I did have dstack in debug print mode so I have the logs leading up to this if those would help.
Edit adding context:
Also running Dstack 0.19.2
This was on a newly created instance so a fresh download of the image.
jvstme commentedon Apr 24, 2025
@ASmedberg-woolpert, thanks for sharing, this looks like the same problem. I think we have enough details to investigate this, just need to find the time to finally get to it. Until the bug is fixed, restarting the run should help
github-actions commentedon Jun 7, 2025
This issue is stale because it has been open for 30 days with no activity.
github-actions commentedon Jun 21, 2025
This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.
jvstme commentedon Jun 30, 2025
Reproduced again on AWS g4dn.xlarge in eu-west-3:
r4victor commentedon Jul 3, 2025
Easier to reproduce when running many runs: I often get 1-2 runs fail with this error when provisioning 100 runs (aws, t3.large).
ASmedberg-woolpert commentedon Jul 10, 2025
I don't have direct numbers to back this up, but the size of the docker image being loaded seems to have an effect: I was using a 12GB image and was seeing this failure fairly frequently (I think more than 1 in 10) but after switching to a 7GB image, I'm seeing it much less frequently (maybe 1 in 20-30). These are all very fuzzy numbers though, so take with a grain of salt.
This all is being run on two dstack servers, one with an AWS back end, and one with an Azure backend. The types of machines vary given our internal quotas.
[shim] Don't check image downloaded size
[shim] Don't check image downloaded size (dstackai#2903)