Skip to content

Add the vendor VFIO vGPU device backend - #364

Open
yummybomb wants to merge 10 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend
Open

Add the vendor VFIO vGPU device backend#364
yummybomb wants to merge 10 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 2 of the vendor VFIO vGPU stack (generalize-vgpu-devicehypervisor-livenessthisvendor-vfio-vgpu). Self-contained in lib/devices + lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).

Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:

  • Discovery & placement — profile discovery from the capacity-dependent creatable_vgpu_types catalogs, least-loaded-GPU VF selection, create/verify/rollback.
  • Release guards — unlike mdev (fresh UUID per assignment), vendor VFIO reuses the same VF path across assignments, so a stale release could clear a later owner's vGPU. Release is guarded by an in-process owner map (covers the window before QEMU opens the device) and an open-VFIO-handle scan (refuses to clear a VF a running VM holds).
  • Reconciliation — clears orphaned assignments on startup, skips VFs in the caller-supplied protected set, and fails closed (skips vendor VFIO entirely) when that set is unavailable, while mdev reconciliation still runs.
  • Integration test — branched by discovered framework, extended to cover release on stop and reacquisition on start.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/resources/ pass

Note

High Risk
Changes sysfs vGPU assignment/release and GPU slot reporting on production hosts; vendor VFIO reuse of VF paths and reconciliation/proc scanning can mis-release or leave orphans if guards fail.

Overview
Adds NVIDIA vendor VFIO vGPU support for hosts where profiles are assigned by writing type IDs to a VF’s nvidia/current_vgpu_type (instead of mdev), and routes all vGPU operations through a single framework-aware path.

Discovery and APIs: DiscoverVGPU() picks mdev when usable VFs exist under mdev_bus, otherwise vendor VFIO when creatable_vgpu_types is present; DetectHostGPUMode is removed. CreateVGPU, DestroyVGPU, ListGPUProfiles, and ReconcileVGPUs dispatch on VGPUFramework. Instance metadata and /resources use GPUDevicePath / framework fields; GetGPUStatus takes context and avoids reporting passthrough when vGPU discovery fails on a vGPU host.

Vendor VFIO behavior: Profile listing uses per-GPU capacity from creatable_vgpu_types; create verifies and rolls back on failure; destroy is guarded by an in-process owner map (InstanceID) and open VFIO handle scans; reconciliation clears orphans while honoring protected device paths.

Lifecycle: VGPUAssignment now carries InstanceID through create/start/stop/release. The vGPU integration test branches on framework (mdev release/reacquire on stop vs vendor VFIO retaining assignment) and adds sysfs helpers plus vendor-specific cleanup.

Docs in GPU.md describe both frameworks, lifecycle, rollback, and troubleshooting.

Reviewed by Cursor Bugbot for commit 78801e7. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 2bb8e86 to 7fc3b49 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7fc3b49 to f661e63 Compare August 6, 2026 19:40
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from f661e63 to d1207d0 Compare August 7, 2026 14:02
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from d1207d0 to 5b47670 Compare August 7, 2026 15:04
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 5b47670 to ffbf8a0 Compare August 7, 2026 20:52
Comment thread lib/devices/vendor_vfio_linux.go
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are
assigned by writing a type ID to a VF's nvidia/current_vgpu_type and
passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend
behind the existing framework dispatch: profile discovery from the
capacity-dependent creatable catalogs, least-loaded VF placement,
create/verify/rollback, and release.

Because the same VF path is reused across assignments (unlike mdev
UUIDs), release is guarded: an in-process owner map covers the window
before QEMU opens the device, and an open-VFIO-handle scan refuses to
clear a VF a running VM still holds. Reconciliation clears orphaned
assignments on startup, skipping VFs protected by the caller and
failing closed when the protected set is unavailable.

Branch the vGPU integration test by discovered framework and extend it
to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from ffbf8a0 to 78801e7 Compare August 8, 2026 01:05

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 78801e7. Configure here.

Comment thread integration/vgpu_test.go
assert.Equal(t, inst.GPUFramework, stopped.GPUFramework, "assignment framework should be retained on stop")
assert.Equal(t, inst.GPUDevicePath, stopped.GPUDevicePath, "assignment metadata should be retained on stop")
assertVGPUAssigned(t, stopped.GPUFramework, stopped.GPUDevicePath)
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vendor stop test expects retention

Medium Severity

The vendor VFIO branch expects stop to keep assignment metadata and sysfs state, but CreateVGPU records the instance in the owner map and StopInstance calls releaseStoredVGPU with that same InstanceID. After the VMM exits, DestroyVGPU clears current_vgpu_type and metadata. That matches GPU.md release-on-stop behavior, so this assertion fails on vendor VFIO hosts and skips the release/reacquire coverage described in the PR.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 78801e7. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant