Skip to content

UCI rigs cannot queue behind a contended device — DeviceLock stops extending on the fourth holder handoff #212

Closed

Description

@JC-000

boot_check.py, rig_https_local.py and rig_https_banner.py hardcode their DeviceLock acquire timeout (60 s / 120 s / 120 s) with no environment override.

The mechanism is not an insufficient literal

DeviceLock.acquire is queue-aware. With the default progress_window=60.0, while the holder's PID is alive and its lockfile mtime is fresh, the deadline is re-armed on every poll and timeout is never consulted. Measured hardware-free:

queue-aware (progress_window=60): acquire(timeout=2.0) behind an 8.0s live holder -> True after 8.0s
legacy      (progress_window=None):                                               -> False after 2.0s

So the hardcoded 120 s would have outlasted an 80-minute run behind a single healthy holder.

What defeats it is the handoff chain. _MAX_HOLDER_HANDOFFS = 3 (device_lock.py:166) is the number of holder-identity changes tolerated; on the fourth, extension stops permanently for that acquire and the literal becomes the entire budget. With several lanes cycling one device, that is reached in seconds.

The fix is therefore an env override for the acquire timeout and preserving progress_window — a larger literal alone does not address the handoff chain.

Upstream

  • JC-000/c64-test-harness#232 — the docstring off-by-one ("after 3 handoffs" vs "on the fourth"). Filed 2026-09-07. Note that the ambiguous phrasing propagated into five files before it was caught; this issue's own title was corrected for the same reason.
  • JC-000/c64-test-harness#233 — env default plus a progress callback. Filed 2026-09-07.

Observed

2026-09-07 on U64E 10.43.23.81. A peer's tools/test_ip65_rrnet_hw.py held the lock for a run of the expected 40-80 minute duration. The rigs could not be launched to queue behind it; the workaround was polling device_lock_holder() read-only every 30 s and launching into the gap, which races any newcomer. Later in the same session the device was held by another lane with DeviceLock.peek_queue_depth() reporting 3 waiters, so the multi-lane contention that triggers the handoff chain is routine, not hypothetical.

Second, related: device config does not survive between lanes

A peer took the lock between two of my runs and left the device at CPU Speed 1 MHz with the REU Disabled. The next comb run's boot precompute (~45 s at 48 MHz, ~36 min at 1 MHz) could not finish inside the rig's 120 s boot budget, and the rig failed at boot with the screen stuck on comb precompute - 45s at 48mhz.

That is a device-config failure that reads exactly like a code defect. It was diagnosed only because the re-prep printed its before-state: REU before: enabled=False, turbo before: Manual/1. Project memory records agents repeatedly misdiagnosing this class as "U64E wedged / firmware corruption" and taking unnecessary destructive recovery action.

Every comb run needs a prep immediately before it, not once per session. The prep must read back and report its before-state, or the next occurrence will be misdiagnosed the same way.

Fix status

A fix exists on local branch fix/uci-devicelock-timeout (not pushed to origin at the time of writing).

Activity

  1. JC-000 commented on Sep 7, 2026

    @JC-000
    OwnerAuthor

    Third observed instance of device config not surviving between lanes

    Logged 2026-09-07 during an unrelated measurement sequence on U64E 10.43.23.81, all three within a few hours:

    1. A peer took the lock between two runs and left the device at CPU Speed 1 MHz with REU Disabled. The next comb run's boot precompute (~45 s at 48 MHz, ~36 min at 1 MHz) could not finish inside the rig's 120 s boot budget, and the rig failed with the screen stuck on comb precompute - 45s at 48mhz — a device-config failure that reads exactly like a code defect. Diagnosed only because the prep read back REU before: enabled=False / turbo before: Manual/1.
    2. A later prep again found turbo before: Manual/1 and had to re-set 48 MHz.
    3. Between two consecutive runs of the same three-run sequence, a prep read back REU before: enabled=False — a peer had disabled the REU mid-sequence.

    Consequence for anyone writing a rig here: a prep at the start of a session is not sufficient. Every run that depends on turbo or the REU needs its own prep immediately beforehand, and the prep must read back and log the before-state, not just write the after-state. Instance 1 was only attributable because that read-back existed; without it the natural conclusion is a comb regression.

    This is the same root cause as the issue itself — several lanes cycling one shared device — seen from the configuration side rather than the lock side. It is also why a run's evidence should carry the device state it observed, alongside the PRG hash.

  2. JC-000 commented on Sep 24, 2026

    @JC-000
    OwnerAuthor

    Closing as already fixed on master (1b9b49e). PR #215 (244a2a0) gives every rig one env-overridable acquire budget: acquire_or_raise(..., progress_window=PROGRESS_WINDOW_S) in tools/uci/_device_lock_helper.py, with boot_check, rig_https_local and rig_https_banner all going through acquire_device_lock(. tools/test_device_lock_timeout.py pins it. The harness halves, #232 and #233, are closed, and the device-config half is covered by #197's prepare_device (110e36f). Reopen if a queued rig still times out behind a live holder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions