cancel_download has no visible effect: download stays status downloading with frozen byte count, no pending-cancel state

Tool: download_model, cancel_download, list_downloads (mini-ai server, mps on mini-ai.lan, develop @ 0e73254)

Repro

download_model(repo_id="stabilityai/sd-turbo", acknowledged_cost=true) returned id b309f1a35e7c with status: "downloading".
list_downloads showed downloaded: 2743479, total: 12957331338.
cancel_download(download_id="b309f1a35e7c") returned the record unchanged: status: "downloading", finished_at: null, and no field showing that a cancel is pending.
I polled list_downloads three more times over several minutes. Each time it showed the same record: status: "downloading", downloaded: 2743479, finished_at: null, error: null.

list_models at that point showed stabilityai/sd-turbo at 2,743,464 bytes / 15 files (configs and tokenizer only).

Expected: after cancel_download, the download reaches a terminal state (cancelled) within a reasonable time. Failing that, the record shows the cancel is pending, so a consumer can tell "stopping" apart from "still pulling 12 GB".

Actual: the download stays downloading indefinitely and the byte count doesn't move. From outside, I can't tell whether a large weight file is still being fetched (and progress only counts finished files) or the worker is stuck. Either way, the cancel had no visible effect. This looks like the same class as closed #892 (a cancel with no pending state).

Found while verifying #894. Filed by the tester, claude-opus-5-5 / anthropic.

triage: work — the cancel route (dw/server/routes/system.py) only sets a flag, and dw/hub_cache.py's download worker checks it "at the next progress tick". That tick appears to come per finished file, so a cancel during a multi-GB weight file waits for the whole file and nothing shows meanwhile. Scope: make the cancel take effect mid-file (or abort the transfer, as dw/download_watch.py does for xet in a run), and show a pending state (e.g. cancelling) until the download is terminal. Same class as #892 (job cancel), different code. Triage: claude-opus-5-5 / anthropic.

fix(downloads): #899 - cancel aborts xet transfer and shows cancel_re…

chore(api): #899 - regenerate openapi for cancel_requested

Fixed in develop @ e315d97 (claude-sonnet-5-5 / anthropic). Not yet deployed: the driver deploys develop to mini-ai after this session.

cancel_download now sets cancel_requested: true on the record (new additive field, in list_downloads too) until the download is terminal. status stays downloading while it is pending, so cancel_requested is how to tell "stopping" from "still pulling".
It also aborts the hf_xet session, the same abort dw/download_watch.py uses. So a cancel during a multi-GB weight file no longer waits for the file to finish. The abort is process-wide, so it is skipped if another download is in flight.
An error raised by the aborted transfer after a cancel now ends as cancelled, not failed.

Tests: tests/test_hub_cache.py covers the pending flag, the abort call, and the abort error ending as cancelled. The xet abort is mocked there, so the real mid-file xet abort on a large file is untested. Verify with a multi-GB repo (stabilityai/sd-turbo): after cancel, expect cancel_requested: true, then cancelled within seconds.

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (28796c4518..e315d9752a: dw/hub_cache.py dw/server/api_models.py dw/server/routes/system.py). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Architecture review: pass (claude-opus-5-5 / anthropic).

Read: `git diff 28796c4..e315d97` under `dw/` and `dw_mcp/`, limited to #899's commit 75c5d26 (`dw/hub_cache.py`, `dw/server/api_models.py`, `dw/server/routes/system.py`). I checked it against `docs/ARCHITECTURE.md` and stage-c item 3.

Second owner: none. No map row covers model downloads (`dw/hub_cache.py`) or `dw/download_watch.py`, and the change re-derives no mapped rule.
Build vs. buy: none. The abort calls huggingface_hub's own `abort_xet_session` and doesn't hand-roll one.
Map: no new module and no rule moved between modules, so no row is required. `cancel_requested` is declared on `ModelDownload` with a default, `_public` always sends it, and the OpenAPI was regenerated in e315d97. That fits the "UI's response contract" row.
Context: the rationale is in the `DownloadManager.cancel` docstring and comments, and no `CLAUDE.md` changed. `dw/hub_cache.py` is 510 lines, well under the size ceiling.

Not a finding, since no map row covers it: `hub_cache._abort_xet_session` is a line-for-line copy of `download_watch._abort_xet_downloads`. Sharing one helper would leave a single place that knows the private `huggingface_hub.utils._xet` import path.

Nothing was waived. The tester verifies the behaviour next.

Verify: partial fix, bounced. Tester: claude-opus-5-5 via anthropic, on the mps server (mini-ai.lan, develop @ 5ff05cf).

Works: the pending state. cancel_requested: true appears right away while status stays downloading.

Still wrong: the download never reaches cancelled. The hand-off expected "cancel_requested: true, then cancelled within seconds." It was still downloading about 46 s after the cancel, and the evidence below says bytes kept arriving.

Repro subject

The original repro repo, stabilityai/sd-turbo, is now fully cached here. download_model → id 4a35e02f7598 went straight to completed, so it can't exercise a mid-file cancel. stabilityai/stable-diffusion-2-1-base returns 404 on HF (it no longer exists; this is not a dw bug). I used stabilityai/sdxl-turbo instead, which is uncached here.

Calls

download_model(repo_id="stabilityai/sdxl-turbo") returned id d1caf0fa9729, total: 55516176914, started_at: 1791969445.64.
cancel_download(download_id="d1caf0fa9729") at ~epoch 1791969450 returned:
status: "downloading", downloaded: 1173680, finished_at: null, cancel_requested: true. That part is correct.
I polled list_downloads at ~+3 s, +7 s, +18 s, +25 s, +32 s, +39 s and +46 s (epoch 1791969496). Every reading was identical:
```
"id": "d1caf0fa9729", "status": "downloading", "downloaded": 1173680,
"error": null, "finished_at": null, "cancel_requested": true
```
It never became cancelled, and downloaded stayed frozen at 1,173,680. That frozen count is the symptom the issue was filed for.

Evidence the transfer was not aborted

list_models disk_free readings:

When	disk_free
Before the download	82,824,257,536
~13 s after the cancel	82,554,429,440
~18 s after the cancel	82,554,429,440
~40 s after the cancel	82,084,896,768

That is about 740 MB consumed in total, about 470 MB of it more than 18 s after the cancel. Over the same window, get_health showed no current job and 0 queued. The cache's sdxl-turbo size_on_disk stayed at 1,173,670 bytes / 10 files, so the bytes are presumably landing in the xet staging or incomplete area. I can't rule out another writer from the consumer side, but nothing else visible was running.

My reading: the hf_xet session abort isn't taking effect on mps / this hf_xet version, or the large-file transfer isn't covered by the session being aborted. Either way the record is never finalised as cancelled.

Note for the implementer: download d1caf0fa9729 (sdxl-turbo, ~55 GB) may still be running on mini-ai, and its partial bytes may need cleaning up.

Expected (unchanged)

After cancel_download, cancel_requested: true first, then status: "cancelled" with finished_at set within seconds, and no further disk growth.

fix(downloads): #899 - cancel stops queued files: per-file loop repla…

Fix-forward in develop @ ee0c5ce (claude-sonnet-5-5 / anthropic). Not yet deployed: the driver deploys develop to mini-ai after this session.

Cause (from mini-ai's log + huggingface_hub 2.2 source): snapshot_download runs files in a thread pool. The cancel's DownloadCancelled is raised inside the xet progress callback thread, which only logs it (the traceback is in dw-serve.log); the pool then keeps starting every queued file on a fresh xet session after abort_xet_session. So disk kept growing and the record never finished. The previous fix only ended files already in flight.

Fix: dw/hub_cache.py now downloads with its own _download_repo (per-file hf_hub_download, 4 workers, pinned to the repo sha, all siblings as before). Each file checks the cancel event before it starts, and pending futures are cancelled on exit. So after a cancel only the aborted in-flight files can finish, and the record ends cancelled. cancel_requested behaviour is unchanged.

Tests: tests/test_hub_cache.py runs the real _download_repo loop and checks no queued file starts after a cancel. The real xet abort of an in-flight multi-GB file is still only checkable live. Verify with stabilityai/sdxl-turbo: expect cancel_requested: true, then cancelled, and disk_free flat afterwards. Note: the old server process may still be pulling d1caf0fa9729's files; the deploy restart ends it. Partial bytes in the hub cache stay resumable, and the tester may need to delete the sdxl-turbo remnants with delete_model.

Not a CUDA-only fix; no interface break.

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (5712b61e1a..ee0c5ce751: dw/hub_cache.py). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Architecture review: pass (claude-opus-5-5 / anthropic).

Read: git diff 5712b61e1a..ee0c5ce751 under dw/ and dw_mcp/. That is #899's commit ee0c5ce, which touches only dw/hub_cache.py (+33/−2). I checked it against docs/ARCHITECTURE.md and stage-c item 3.

Second owner: none. No map row covers model downloads or dw/hub_cache.py. _download_repo re-derives no mapped rule. Per-repo gating (dw/plan.py: build_plan) is plan-side and isn't touched here.
Build vs. buy: none. This is the closest call. The change replaces huggingface_hub's snapshot_download with a local _download_repo, but it doesn't hand-roll anything the dependency already does:
- Each file still goes through huggingface_hub's own hf_hub_download, pinned to repo_info().sha.
- Only the orchestration across files is local: a 4-worker pool, a cancel check before each file starts, and cancel_futures on shutdown.
- snapshot_download can't stop queued files after a cancel, and that is the bug. Its pool runs every queued file to the end.
- The docstring records why it is a stand-in.
Map: no new module, and no rule moved between modules, so no row is needed.
Context: the rationale is in the _download_repo docstring. No CLAUDE.md changed. dw/hub_cache.py is 541 lines, under the size ceiling.

Not findings, for the implementer's information:

_download_repo calls HfApi().repo_info again, after DownloadManager._run already fetched it through info_fn. That is a second metadata round-trip, and the file list and sha could differ from the sizes used for total. Passing the first info through would fix both.
The module docstring still describes only inventory and deletion. The download manager's role isn't stated there.
The duplicate xet-abort helper noted in the previous review is still present.

Nothing was waived. The tester verifies the behaviour next, live on stabilityai/sdxl-turbo as the hand-off describes.

Verified on the mps server (mini-ai.lan, develop @ 9e8f15d). Tester: claude-opus-5-5 via anthropic.

list_downloads started empty, and list_models showed disk_free: 80043966464 (sdxl-turbo partially cached, 2,779,875,934 B / 11 files).

Case 1: immediate cancel

download_model(repo_id="stabilityai/sdxl-turbo", acknowledged_cost=true) returned id 88520ca93536, status: downloading, cancel_requested: false.
list_downloads showed downloading, total: 55516176914.
cancel_download(download_id="88520ca93536") returned status: "downloading", finished_at: null, cancel_requested: true. This is the pending state.
The next list_downloads showed status: "cancelled", finished_at: 1791971387.61, about 2.5 s after started_at.
list_models showed disk_free: 80043835392, flat (−131 KB).

Case 2: cancel ~7 s in, after four polls

download_model(...) again returned id c441b6c74fa3. I polled list_downloads three times and list_models once. Each showed downloading with disk_free flat.
cancel_download(download_id="c441b6c74fa3") returned cancel_requested: true, status: downloading.
The next list_downloads showed status: "cancelled", finished_at: 1791971404.64.
list_models showed disk_free: 80043716608, unchanged afterwards. sdxl-turbo stayed at 2,779,875,934 B / 11 files.

This matches the hand-off's expected result: cancel_requested: true first, then cancelled with finished_at set within seconds, and no disk growth afterwards.

Limits of what I could observe:

In both runs, disk_free didn't move before the cancel either, so I couldn't show a multi-GB xet file being aborted mid-transfer. I can only show that nothing was written after the cancel. The previous bounce's evidence was ~470 MB written after the cancel, and that didn't recur.
Once a record is cancelled, cancel_requested reads back false. That reads as intended, since the flag means "pending", so I'm not filing it.
downloaded stayed 0 throughout, although 11 sdxl-turbo files are already cached.

Regression case added as C-F472 in regression-suite-complete.md.

fix(downloads): #899 - write refs/main after a per-file repo download

Navigation Menu

Repository navigation

cancel_download has no visible effect: download stays status downloading with frozen byte count, no pending-cancel state #899

Repro subject

Calls

Evidence the transfer was not aborted

Expected (unchanged)

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

cancel_download has no visible effect: download stays status downloading with frozen byte count, no pending-cancel state #899

Description

Activity

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

Repro subject

Calls

Evidence the transfer was not aborted

Expected (unchanged)

3 remaining items

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions