Integrate vendor VFIO vGPUs into the instance lifecycle - #321
Conversation
199b1d1 to
a93d009
Compare
d893fce to
de4742a
Compare
de4742a to
18e047f
Compare
|
Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design: |
7ef1f8d to
bf21162
Compare
bf21162 to
83c53ea
Compare
83c53ea to
79eef07
Compare
ab2b0e8 to
19b9602
Compare
d92201d to
f177335
Compare
f177335 to
1d875f2
Compare
A vGPU assignment goes stale when its release succeeds but the metadata save does not (or start fails between the release and its first save). The backend's owner map only covers assignments created since the last restart and the VFIO handle scan only covers VMs that have opened the device, so after a restart a stale release could still clear a VF during another live instance's pre-open boot window. Consult live instance metadata on every release: when another instance with a live hypervisor process claims the same device path, drop the stale metadata without touching the device. Tag assignments with the owning instance ID, persist the assignment before booting a started instance, and retain assignment metadata when rollback release fails in create and start so later release paths can still find the device.
Startup reconciliation protects the VFs of instances whose hypervisor survived the restart, verified by socket ownership so a reused PID cannot hold a VF. The inventory behind that protected set must not silently skip unreadable metadata: a skipped live claimant would leave its VF unprotected during the pre-VFIO-open boot window. Add ListInstancesForReconcile, which fails on any unreadable metadata, and skip vendor VFIO reconciliation when the inventory is unavailable while keeping mdev reconciliation running.
1d875f2 to
23fb2f5
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 23fb2f5. Configure here.
| } | ||
| } | ||
| return false, nil | ||
| } |
There was a problem hiding this comment.
Release claim check skips bad metadata
High Severity
The vgpuAssignmentClaimedByLiveInstance check uses listInstances, which skips unreadable metadata. This differs from the fail-closed ListInstancesForReconcile used by startup protection. If a live instance's metadata is unreadable, the claim check can miss, leading releaseStoredVGPU to destroy a vGPU still in use by that live instance.
Reviewed by Cursor Bugbot for commit 23fb2f5. Configure here.


Summary
Top layer of the vendor VFIO vGPU stack (
generalize-vgpu-device#322 ←hypervisor-liveness#363 ←vendor-vfio-backend#364 ← this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.Testing
go build ./...,go vet ./...cleango test -racepasses forlib/instancestargeted suites,lib/devices,lib/resources,lib/builds(TestCreateInstanceWithNetworkneeds image pulls + iptables and fails in this environment on the unmodified base as well)Note
High Risk
Changes GPU release and startup reconciliation for vendor VFIO; incorrect claim detection or protected-set building could detach GPUs from live VMs or leave orphans.
Overview
Wires vendor VFIO vGPUs into create/start/stop/delete with safety around reused VF paths: releases go through
releaseStoredVGPUon the manager, tag assignments with instance ID, and skipDestroyVGPUwhen another live instance (socket + PID liveness) already claims the sameGPUDevicePath—only clearing stale metadata.Durability on failure: failed create cleanup can retain GPU fields on disk instead of wiping the guest dir; start persists metadata after vGPU creation and on failed rollback destroy; create/start cleanup paths mirror that retention behavior.
Startup reconcile: API startup builds a protected set of VF paths from
ListInstancesForReconcile(strict inventory—invalid metadata fails the list) and passes it todevices.ReconcileVGPUsinstead of bare mdev reconcile; if listing fails, vendor VFIO reconciliation is skipped (warn) while other paths still run.Adds
ListInstancesForReconcileon the instance manager (vs list skipping bad metadata) and tests for stale-claim delete, retained failed-create metadata, and reconcile list strictness.Reviewed by Cursor Bugbot for commit 23fb2f5. Bugbot is set up for automated code reviews on this repo. Configure here.