Add the vendor VFIO vGPU device backend - #364
Conversation
2bb8e86 to
7fc3b49
Compare
7fc3b49 to
f661e63
Compare
f661e63 to
d1207d0
Compare
d1207d0 to
5b47670
Compare
5b47670 to
ffbf8a0
Compare
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend behind the existing framework dispatch: profile discovery from the capacity-dependent creatable catalogs, least-loaded VF placement, create/verify/rollback, and release. Because the same VF path is reused across assignments (unlike mdev UUIDs), release is guarded: an in-process owner map covers the window before QEMU opens the device, and an open-VFIO-handle scan refuses to clear a VF a running VM still holds. Reconciliation clears orphaned assignments on startup, skipping VFs protected by the caller and failing closed when the protected set is unavailable. Branch the vGPU integration test by discovered framework and extend it to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
…nership through releases
ffbf8a0 to
78801e7
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 78801e7. Configure here.
| assert.Equal(t, inst.GPUFramework, stopped.GPUFramework, "assignment framework should be retained on stop") | ||
| assert.Equal(t, inst.GPUDevicePath, stopped.GPUDevicePath, "assignment metadata should be retained on stop") | ||
| assertVGPUAssigned(t, stopped.GPUFramework, stopped.GPUDevicePath) | ||
| }) |
There was a problem hiding this comment.
Vendor stop test expects retention
Medium Severity
The vendor VFIO branch expects stop to keep assignment metadata and sysfs state, but CreateVGPU records the instance in the owner map and StopInstance calls releaseStoredVGPU with that same InstanceID. After the VMM exits, DestroyVGPU clears current_vgpu_type and metadata. That matches GPU.md release-on-stop behavior, so this assertion fails on vendor VFIO hosts and skips the release/reacquire coverage described in the PR.
Reviewed by Cursor Bugbot for commit 78801e7. Configure here.


Summary
Layer 2 of the vendor VFIO vGPU stack (
generalize-vgpu-device←hypervisor-liveness← this ←vendor-vfio-vgpu). Self-contained inlib/devices+lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's
nvidia/current_vgpu_typeand passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:creatable_vgpu_typescatalogs, least-loaded-GPU VF selection, create/verify/rollback.Testing
go build ./...,go vetcleango test -race ./lib/devices/ ./lib/resources/passNote
High Risk
Changes sysfs vGPU assignment/release and GPU slot reporting on production hosts; vendor VFIO reuse of VF paths and reconciliation/proc scanning can mis-release or leave orphans if guards fail.
Overview
Adds NVIDIA vendor VFIO vGPU support for hosts where profiles are assigned by writing type IDs to a VF’s
nvidia/current_vgpu_type(instead of mdev), and routes all vGPU operations through a single framework-aware path.Discovery and APIs:
DiscoverVGPU()picks mdev when usable VFs exist undermdev_bus, otherwise vendor VFIO whencreatable_vgpu_typesis present;DetectHostGPUModeis removed.CreateVGPU,DestroyVGPU,ListGPUProfiles, andReconcileVGPUsdispatch onVGPUFramework. Instance metadata and/resourcesuseGPUDevicePath/ framework fields;GetGPUStatustakescontextand avoids reporting passthrough when vGPU discovery fails on a vGPU host.Vendor VFIO behavior: Profile listing uses per-GPU capacity from
creatable_vgpu_types; create verifies and rolls back on failure; destroy is guarded by an in-process owner map (InstanceID) and open VFIO handle scans; reconciliation clears orphans while honoring protected device paths.Lifecycle:
VGPUAssignmentnow carriesInstanceIDthrough create/start/stop/release. The vGPU integration test branches on framework (mdev release/reacquire on stop vs vendor VFIO retaining assignment) and adds sysfs helpers plus vendor-specific cleanup.Docs in
GPU.mddescribe both frameworks, lifecycle, rollback, and troubleshooting.Reviewed by Cursor Bugbot for commit 78801e7. Bugbot is set up for automated code reviews on this repo. Configure here.