Skip to content

Run checks and spm-tests on the owner's Mac when it's idle - #1810

Merged
r3dbars merged 10 commits into
mainfrom
claude/self-hosted-mac-runner-1qxwmv
Sep 24, 2026
Merged

r3dbars merged 10 commits into
mainfrom
claude/self-hosted-mac-runner-1qxwmv

Conversation

@r3dbars

@r3dbars r3dbars commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Before: every PR's Swift CI run needs 3 of the account's 5 hosted macOS slots (checks, spm-tests, app-build). With several PRs open, most runs sit in the queue.

After: checks and spm-tests run on the owner's Mac whenever it's free, each in a fresh throwaway macOS VM. Only app-build needs a hosted slot. Both jobs go to hosted macos-26, exactly like today, whenever the Mac is:

  • busy, asleep, or on battery
  • on a call
  • paused, low on disk, or not set up

Until someone runs scripts/ci/mac-runner.sh install on the Mac, nothing changes. With no heartbeat, everything stays hosted.

How:

Routing. A new Linux job, pick-runner (scripts/ci/pick-ci-runner.py, actions: read), picks the Mac only when all of these hold:

  • the event is a push, a dispatch, or a same-repo PR
  • the MAC_RUNNER_HEARTBEAT repo variable is a bare timestamp under 60s old
  • no other run already has a Mac job queued or running

Otherwise it picks hosted. Any error means hosted too, and the public log never says why the Mac is busy.

The Mac service (scripts/ci/mac-runner.sh serve) is a launch agent in the owner's account.

  • It boots a VM only when a job is actually waiting for it.
  • It clones a stopped golden Tart VM at low priority, then hands it a one-job JIT runner registration through a read-only share once it has booted.
  • It deletes the VM afterwards, and backs off if a VM fails to boot.
  • To keep API use low, it remembers runs that went hosted, and polls slowly when no job can be on its way.

Waiting jobs always run. A job already sent to the Mac runs even if the owner pauses, unplugs, or joins a call. During a call it just drops to background priority.

Nothing waits forever.

  • If the Mac can't start a waiting job for 15 minutes, it re-runs that run on hosted.
  • A new mac-runner-sweep.yml workflow runs main's copy of the script every 30 minutes, with actions: write. It does the same for runs stuck behind a Mac that went quiet (asleep, off, or a dead service). It does nothing until the heartbeat exists.

Job isolation.

  • Inside the VM, a pf firewall loaded at boot allows the internet but blocks the host Mac, other VMs, Tailscale/CGNAT, the LAN, and all IPv6.
  • The job's user has no admin rights.
  • Building the golden image probes those addresses as the job user and requires pf's counters to show the blocks. It also proves the lockdown and the fork-refusing job-started hook.
  • The owner's gh login never enters a VM.

Fork PRs stay off the Mac three ways:

  • install sets the "approve all outside contributors" policy.
  • pick-runner never routes them.
  • The in-VM hook refuses them.

Pinned downloads. Tart comes from the version and checksum pinned in scripts/vm/transcripted-vm.sh. It's installed into ~/.transcripted-ci and never touches ~/.transcripted-vm. The CI image is pinned by digest.

CI wiring.

  • build-and-test also requires pick-runner.
  • repo-hygiene runs both self-tests.
  • Tests/CIWorkflowContractTests.swift pins that only checks and spm-tests can take the Mac.

Docs: docs/self-hosted-mac-runner.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw

GitHub allows 5 concurrent hosted macOS jobs and each Swift CI run needs 3,
so PRs queue. A new pick-runner job sends checks and spm-tests to self-hosted
runners labelled transcripted-mac when the Mac's heartbeat variable says a
runner is idle, and to hosted macos-26 otherwise. app-build stays hosted.

Fork PRs never reach the Mac: pick-runner keeps them hosted, and the Mac's
job-started hook refuses any job that is not push, workflow_dispatch, or a
same-repo pull_request, even if a fork rewrites the workflow.

scripts/ci/mac-runner.sh installs, pauses, and removes the runners and the
heartbeat launchd agent. Both scripts self-test in repo-hygiene.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw
@r3dbars r3dbars self-assigned this Sep 24, 2026
Review of #1810 found the runners ran as the owner's own user, next to his
gh admin login, keychain, and data, and matched hardware-smokes' labels.

- install now creates a standard "Transcripted CI" account (own group, not
  in staff/admin, 700 home) and registers one runner there with only the
  transcripted-mac label. A root-owned launch agent starts it in that
  account's login session at Nice 10. The owner's gh login is used only by
  install/uninstall and the heartbeat, which runs as the owner.
- The job-started hook and launcher live root-owned in /Library/TranscriptedCI.
  The hook starts from an empty environment and checks the payload's repo
  for every event, not just pull_request.
- install refuses while any other runner is registered on the repo, and
  verifies the runner download's SHA-256.
- pick-runner (now Python) also counts Mac jobs already queued or running
  in other runs via a read-only token and goes hosted if there are any.
  The heartbeat runs every 20s with a 60s window, reports busy/mic/offline/
  battery/paused, and keeps the Mac awake while a job runs.
- The Mac's deps cache uses its own deps-mac- key and saves only from main.
- Stall diagnostics only sample this account's xctest and only print recent
  crash reports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw
The re-check of the separate-account design found more holes: a root
chown -R over a home the CI account controls, staff membership through
nested groups, and a runner the CI account could tamper with for later jobs.
All of them come from running jobs on the host at all.

Now the owner's launch agent (mac-runner.sh serve) clones a stopped golden
VM per job, hands it a one-job just-in-time runner registration through a
read-only shared folder, boots it with no host audio or clipboard, waits for
the job, and deletes the VM and the registration. Nothing runs as root, no
macOS account is created, and the owner's gh login never enters a VM.

Building the golden VM installs a SHA-256-checked runner and the job-started
hook, then proves the hook accepts a same-repo push and refuses a fork PR
before keeping the image. Tart comes from the pinned, checked install in
scripts/vm/transcripted-vm.sh. The golden VM is rebuilt every 14 days.

The heartbeat now reads the runner's state from the API instead of pgrep,
and the install EXIT trap and HOME-less hook environment bugs are gone with
the old code. The Mac's spm-tests may restore hosted deps entries and uses
the Tools SwiftPM cache again, since each VM starts empty.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw
- install: install the Tart release pinned in transcripted-vm.sh straight
  into ~/.transcripted-ci instead of calling that script, whose home guard
  refused this folder. self-test proves the pin still parses.
- Each job VM gets a pf firewall at boot: internet only, no host, other VMs
  or LAN. The job user loses admin rights. The golden build proves both.
- No idle VM: the service boots one only when a Swift CI job is queued for
  the Mac, at low priority, and counts VM slots first. A job already sent
  to the Mac always runs; during a call it drops to background priority.
- Back off when a VM fails to boot; mint a registration only after boot.
- Check free disk before each job; pin the CI image by digest and replace
  it when the pin changes; build the golden image beside the old one.
- A waiting job the Mac can't start for 15 minutes is re-run on hosted, and
  pick-runner does the same for runs stuck behind a Mac that went quiet.
- Heartbeat words carry a timestamp; public logs no longer say why the Mac
  is busy. Hard per-VM deadline, safer uninstall, working log rotation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw
- The service remembers runs whose jobs went hosted and stops asking about
  them, pages past 30 runs, polls every 2 minutes when no job can be on its
  way, and stops taking runs when the owner's gh quota runs low.
- The golden build probes the Mac's ports, Tailscale, LAN ranges and IPv6
  as the job user, and requires pf's own counters to show the blocks. It
  also logs which uid tart exec runs as and checks the admin group.
- Rerouting runs stuck behind a quiet Mac moves out of pick-runner (back to
  actions: read) into mac-runner-sweep.yml, which runs main's copy every 30
  minutes, so a dead service can't leave a job waiting 24 hours.
- Only renice the VM process when exactly one new one appeared.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw
…c-runner-1qxwmv

# Conflicts:
#	.github/workflows/repo-hygiene.yml
- A runner that drops off GitHub mid-job frees the Mac after 10 minutes
  instead of the full job limit.
- A registration that never shows up on GitHub counts as a failed VM (back
  off), not a finished job.
- Rerouting force-cancels a run whose cancel stalls and retries the re-run.
- install and rebuild park the service between jobs through a STOP file
  before stopping it, and restart it if they fail.
- The golden build moves any root daemon program the job user could replace
  (e.g. under /opt/homebrew) somewhere only root can write.
- DNS rule is IPv4 only; one heartbeat write per loop while a job waits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw
- While parked for install/rebuild, the service re-runs waiting jobs first
  and only then reports "stopped", so it can't be stopped mid-reroute.
- install/rebuild set their restart trap before parking the service;
  resume clears a leftover park and status shows it.
- lock-daemons also refuses paths the job user owns, checks moved
  binaries' libraries, DYLD_/PATH settings, login hooks and periodic
  scripts, and gives each moved program a unique name.
- A run is only remembered as settled once pick-runner has finished and
  both jobs have their runner labels.
- Heartbeat keeps ticking while waiting on a registration or a reroute.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cgn9ZKF6TdiobvwYDDmDxw
r3dbars pushed a commit that referenced this pull request Sep 24, 2026
Four open PRs (#1791, #1803, #1810, #1812) append steps to the inline
repo-hygiene workflow, so replacing those steps would conflict and a lazy
resolution would silently drop their checks. Keep the original steps
byte-for-byte and insert a single "Linux checks (scripts/dev/linux-checks.sh)"
step right after Checkout, away from where the other PRs append. The
overlap in runtime is intentional.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GhNAWeNXuLGdq4ioddevdp
@r3dbars
r3dbars marked this pull request as ready for review September 24, 2026 04:08
@r3dbars
r3dbars merged commit 471a3da into main Sep 24, 2026
8 checks passed
@r3dbars
r3dbars deleted the claude/self-hosted-mac-runner-1qxwmv branch September 24, 2026 04:08
claude Bot pushed a commit that referenced this pull request Sep 24, 2026
Sources/CLAUDE.md: keep the one-meeting Boost wording and add Apple Speech
to the model list.

Also time the "can't hear the call" stretch by wall clock. recordingDuration
reads 0 on an unexpected stop, so a long loss there would not be marked
degraded (#1811 hit the same reset for meeting length).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgXMcLRGBtiSiybdgyD1VJ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants