Skip to content

A .clearing-* staging directory orphaned by a killed clear is never swept #839

Closed

Description

@dkackman

From the 2026-10-11 develop review.

disk_caches.clear() renames the root's contents into <root>/.clearing-<pid>-<rand>/ and then rmtrees it. If the server dies mid-rmtree the staging directory survives: sweep() ignores it, _walk counts its files as live cache bytes and newest mtime, and mark_root treats the root as populated. Only the next full clear removes it.

Failure: server killed during the rmtree of a 20 GB triton cache -> GET /api/caches and the Caches panel report those 20 GB as the cache's size until someone clears again.

Fix direction: at mark_roots() (server start) and in _reason()/scan, remove any leftover .clearing-* under a marked root; _walk skips the prefix.

Activity

  1. dkackman commented on Oct 12, 2026

    @dkackman
    OwnerAuthor

    triage: batch with #838, #839, #840, #841 — all in dw/disk_caches.py and the /api/caches route (clear lock, staging sweep, scan cost, mark_root logging); one fix, one server restart. (claude-opus-5-5 via anthropic)

  2. added a commit that references this issue on Oct 12, 2026
  3. dkackman commented on Oct 12, 2026

    @dkackman
    OwnerAuthor

    Fixed in 5dc3ff4 on develop: _walk skips top-level .clearing-*, and leftovers are removed at mark_roots() (server start) and inside the lock at the next clear/entry delete. Tests: scan ignores a leftover; mark_roots and clear remove it.

    Batch #838/#839/#840/#841, one commit. Tests pass in the source checkout (tests/test_disk_caches.py, test_server.py, test_api_contract.py; OpenAPI doc regenerated for the changed route docstrings). The full suite's 18 other failures are Kandinsky6 tests that fail on develop without this change. Not deployed by me: the driver deploys develop to mini-ai before the tester runs. Server restart needed; no breaking MCP change. Hard to verify over MCP: the race itself (window is milliseconds); verifiable is DELETE /api/caches/ still returning 200/409/400 as before and get_server_info caches summary still present.

    — claude-sonnet-5-5 via anthropic

  4. dkackman commented on Oct 12, 2026

    @dkackman
    OwnerAuthor

    Verified on the mini-ai server (mps on mini-ai.lan), develop @ 68ba49b, which includes 5dc3ff4. The verification only covers the part an MCP consumer can reach.

    What I ran: get_server_info → caches is present with all three entries and sane values:

    • inductor: 0 bytes / 0 files, enabled, clearable
    • triton: 0 / 0, disabled, not clearable
    • quantized: 0 / 0, entry_count: 0

    The server came up on the new mark_roots() startup sweep and the scan still reports normally. Nothing in the cache summary changed shape or regressed.

    What I could not observe: the fix itself. That means an orphaned .clearing-* directory being skipped by _walk and removed at mark_roots() or the next clear. Reproducing it needs either a server killed partway through an rmtree or a hand-planted staging directory under a cache root. A consumer can do neither: there is no MCP tool that clears caches (that is DELETE /api/caches/<name>, and REST is outside my fence), and I can't touch the box. For that behaviour I'm relying on the implementer's unit tests (scan ignores a leftover; mark_roots and clear remove it), not on anything I saw.

    Closing on that basis. If a .clearing-* leftover is ever seen inflating caches[].bytes, that is new evidence for a new issue referencing this one.

    No regression case added: the only thing I could observe is that the caches summary is present, and existing cases already cover that.

    — claude-opus-5-5 via anthropic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backend:sharedbugSomething isn't workingowner:testerTester's turn to actstatus:verifiedTester confirmed the fix via a real MCP calltarget:mini-aiFound against a non-lem server (DW_TARGET=mini-ai); Don triagesverified-on:mps

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions