Harden hypervisor process liveness checks - #363
Open
yummybomb wants to merge 8 commits into
Open
Conversation
This was referenced Aug 6, 2026
yummybomb
marked this pull request as ready for review
August 6, 2026 19:20
yummybomb
force-pushed
the
hypeship/hypervisor-liveness
branch
from
August 6, 2026 19:26
f9e6580 to
3c8b92f
Compare
yummybomb
force-pushed
the
hypeship/hypervisor-liveness
branch
from
August 6, 2026 19:40
3c8b92f to
915f0ef
Compare
yummybomb
force-pushed
the
hypeship/hypervisor-liveness
branch
from
August 7, 2026 14:02
915f0ef to
a187984
Compare
yummybomb
force-pushed
the
hypeship/hypervisor-liveness
branch
2 times, most recently
from
August 7, 2026 20:52
76b9f42 to
78fc483
Compare
kill(pid, 0) returning EPERM means the process exists but cannot be signaled, and a zombie PID passes a bare kill(0) probe. Export the EPERM-aware, zombie-filtering processExists helper so every hypervisor liveness check shares one definition.
After a hypeman restart the hypervisor is not our child, so Wait4 returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual process exit in that case.
A bare liveness probe treats any process that reused a stored hypervisor PID as the owning VMM. Require the PID to own the instance's hypervisor socket on Linux before reporting it alive.
Accepted server-side sockets appear in /proc/net/unix with the same bound path as the listener, so any connected API client made socketRefForPath report multiple inodes and pid-reuse protection fell back to unconfirmed while the control socket was in use. Only entries with __SO_ACCEPTCON identify the owning process; duplicate listeners from unlink-and-rebind still resolve as unconfirmed.
yummybomb
force-pushed
the
hypeship/hypervisor-liveness
branch
from
August 8, 2026 01:05
78fc483 to
f8fbe79
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f8fbe79. Configure here.
| log.DebugContext(ctx, "hypervisor process killed", "instance_id", inst.Id, "pid", pid) | ||
| break | ||
| } | ||
| } |
There was a problem hiding this comment.
Socket unlinked after skipped kill
High Severity
When socket ownership cannot be confirmed or an unconfirmed PID mismatch skips the kill, killHypervisor still calls os.Remove on SocketPath. That unlinks a live VMM’s control socket after the new logic deliberately refused to kill it, turning an intended recoverable leak into an unmanageable orphan.
Reviewed by Cursor Bugbot for commit f8fbe79. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Layer 1 of the vendor VFIO vGPU stack (
generalize-vgpu-device← this ←vendor-vfio-backend←vendor-vfio-vgpu). Pure hypervisor-process hardening with no vGPU-specific code; reviewable in isolation.The upper layers guard vGPU release decisions on "is this instance's hypervisor still alive", so the liveness answer has to be trustworthy first:
ProcessExists— one exported, EPERM-aware, zombie-filtering definition instead of scattered barekill(pid, 0)probes. EPERM means the process exists but cannot be signaled; treating it as dead would be wrong.Wait4returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual exit in that case.HypervisorProcessExists: verify socket ownership — a bare PID probe treats any process that reused a stored hypervisor PID as the owning VMM. On Linux, require the PID to own the instance's hypervisor socket before reporting it alive.Testing
go build ./...,go vetcleango test -race ./lib/instances/targeted suites pass (TestCreateInstanceWithNetworkrequires image pulls + iptables and fails in this environment on the unmodified base as well)Note
Medium Risk
Changes delete-time SIGKILL targeting and hypervisor liveness signals; the design favors skipping kills over killing unrelated processes, but mis-resolved socket ownership could still leave a leaked VMM or kill the wrong owner in edge cases.
Overview
Hardens how hypeman decides whether a hypervisor is still alive and which PID to tear down on delete, so stale metadata after a restart cannot point SIGKILL at the wrong process.
Linux
ResolveProcessPIDnow returns aconfirmedflag when the PID comes from socket inode / FD ownership (vs cmdline fallback). Socket lookup in/proc/net/unixonly treats__SO_ACCEPTCONlisteners as the owner, ignores accepted connections that share the path, errors on ambiguous duplicate listeners, and surfaces FD scan failures instead of silently falling through.Instance layer: unexported
processExistsbecomes exportedProcessExists(EPERM-aware, zombie filtering on Linux). NewHypervisorProcessExistsrequires the stored PID to own the instance control socket when ownership can be confirmed.killHypervisoronly kills when ownership is confirmed (or on non-Linux / no socket path); on a confirmed mismatch it kills the socket owner; when ownership is unknown it skips the kill. The post-SIGKILL wait loop keeps polling until the process is gone whenWait4returnsECHILD(hypervisor not a child after hypeman restart).Integration tests cover reused PIDs, missing sockets, and socket rebound scenarios.
Reviewed by Cursor Bugbot for commit f8fbe79. Bugbot is set up for automated code reviews on this repo. Configure here.