Skip to content

Integrate vendor VFIO vGPUs into the instance lifecycle - #321

Open
yummybomb wants to merge 2 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu
Open

Integrate vendor VFIO vGPUs into the instance lifecycle#321
yummybomb wants to merge 2 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu

Conversation

@yummybomb

@yummybomb yummybomb commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Top layer of the vendor VFIO vGPU stack (generalize-vgpu-device #322hypervisor-liveness #363vendor-vfio-backend #364this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:

  • Live-instance claim guard on release — an assignment goes stale when its release succeeds but the metadata save does not (or start fails between the release and its first save). Since vendor VFIO reuses the same VF path across assignments, honoring a stale claim could clear a VF out from under the live instance that owns it now. Every release consults live instance metadata (liveness verified via Harden hypervisor process liveness checks #363's socket-ownership check) and drops stale metadata without touching the device.
  • Assignment durability — assignments are tagged with the owning instance ID, persisted before booting a started instance, and retained in metadata when rollback release fails in create/start so later release paths can still find the device.
  • Fail-closed startup reconciliation — the protected set of live VFs is built from ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.

Testing

  • go build ./..., go vet ./... clean
  • go test -race passes for lib/instances targeted suites, lib/devices, lib/resources, lib/builds (TestCreateInstanceWithNetwork needs image pulls + iptables and fails in this environment on the unmodified base as well)

Note

High Risk
Changes GPU release and startup reconciliation for vendor VFIO; incorrect claim detection or protected-set building could detach GPUs from live VMs or leave orphans.

Overview
Wires vendor VFIO vGPUs into create/start/stop/delete with safety around reused VF paths: releases go through releaseStoredVGPU on the manager, tag assignments with instance ID, and skip DestroyVGPU when another live instance (socket + PID liveness) already claims the same GPUDevicePath—only clearing stale metadata.

Durability on failure: failed create cleanup can retain GPU fields on disk instead of wiping the guest dir; start persists metadata after vGPU creation and on failed rollback destroy; create/start cleanup paths mirror that retention behavior.

Startup reconcile: API startup builds a protected set of VF paths from ListInstancesForReconcile (strict inventory—invalid metadata fails the list) and passes it to devices.ReconcileVGPUs instead of bare mdev reconcile; if listing fails, vendor VFIO reconciliation is skipped (warn) while other paths still run.

Adds ListInstancesForReconcile on the instance manager (vs list skipping bad metadata) and tests for stale-claim delete, retained failed-create metadata, and reconcile list strictness.

Reviewed by Cursor Bugbot for commit 23fb2f5. Bugbot is set up for automated code reviews on this repo. Configure here.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 199b1d1 to a93d009 Compare July 28, 2026 21:32
@yummybomb
yummybomb changed the base branch from main to hypeship/generalize-vgpu-device July 28, 2026 21:32
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch 4 times, most recently from d893fce to de4742a Compare July 29, 2026 15:15
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from de4742a to 18e047f Compare July 29, 2026 16:09
@yummybomb

yummybomb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design:

@yummybomb
yummybomb marked this pull request as ready for review August 5, 2026 19:47
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 7ef1f8d to bf21162 Compare August 5, 2026 19:57
Comment thread lib/instances/start.go
Comment thread integration/vgpu_test.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from bf21162 to 83c53ea Compare August 5, 2026 20:24
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 83c53ea to 79eef07 Compare August 5, 2026 20:39
Comment thread cmd/api/main.go
Comment thread lib/devices/vendor_vfio_linux.go
Comment thread lib/instances/query.go Outdated
Comment thread lib/instances/create.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from ab2b0e8 to 19b9602 Compare August 6, 2026 17:08
Comment thread lib/instances/start.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from d92201d to f177335 Compare August 6, 2026 19:06
@yummybomb
yummybomb changed the base branch from hypeship/generalize-vgpu-device to hypeship/vendor-vfio-backend August 6, 2026 19:08
@yummybomb yummybomb changed the title Support vendor VFIO vGPU devices Integrate vendor VFIO vGPUs into the instance lifecycle Aug 6, 2026
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from f177335 to 1d875f2 Compare August 6, 2026 19:26
A vGPU assignment goes stale when its release succeeds but the
metadata save does not (or start fails between the release and its
first save). The backend's owner map only covers assignments created
since the last restart and the VFIO handle scan only covers VMs that
have opened the device, so after a restart a stale release could still
clear a VF during another live instance's pre-open boot window.
Consult live instance metadata on every release: when another instance
with a live hypervisor process claims the same device path, drop the
stale metadata without touching the device.

Tag assignments with the owning instance ID, persist the assignment
before booting a started instance, and retain assignment metadata when
rollback release fails in create and start so later release paths can
still find the device.
Startup reconciliation protects the VFs of instances whose hypervisor
survived the restart, verified by socket ownership so a reused PID
cannot hold a VF. The inventory behind that protected set must not
silently skip unreadable metadata: a skipped live claimant would leave
its VF unprotected during the pre-VFIO-open boot window. Add
ListInstancesForReconcile, which fails on any unreadable metadata, and
skip vendor VFIO reconciliation when the inventory is unavailable
while keeping mdev reconciliation running.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 1d875f2 to 23fb2f5 Compare August 6, 2026 19:40

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 23fb2f5. Configure here.

Comment thread lib/instances/vgpu.go
}
}
return false, nil
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Release claim check skips bad metadata

High Severity

The vgpuAssignmentClaimedByLiveInstance check uses listInstances, which skips unreadable metadata. This differs from the fail-closed ListInstancesForReconcile used by startup protection. If a live instance's metadata is unreadable, the claim check can miss, leading releaseStoredVGPU to destroy a vGPU still in use by that live instance.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 23fb2f5. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant