Quarantine unhealthy vGPU VFs via a persisted health store - #462
Quarantine unhealthy vGPU VFs via a persisted health store#462yummybomb wants to merge 6 commits into
Conversation
182853a to
34d27c2
Compare
-->
✱ stlc build✅ go code · compare
✅ python code · compare
✅ typescript code · compare
Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
Build metadata
This comment is auto-generated by stlc and is kept up to date as you push. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2bea9f0. Configure here.
|
Cleanup pass (deslop / test dedup), no behavior changes:
The persist-failure/rollback/retry test family was left intact — each pins a distinct branch fixed by Verified: |
Add a VF health store persisted at <data-dir>/gpu/vf-health.json: init failures reported against a VF are tallied per instance assignment, and once failures accumulate from gpu.vf_quarantine_threshold distinct assignments (default 2) the VF is quarantined. Quarantined VFs are excluded from placement and advertised profile availability, cards with quarantined VFs are deprioritized, and selection among equivalent free VFs is randomized. An exact-assignment success report clears the match and older tallies and rescinds that assignment's quarantine. An unreadable or invalid state file fails closed: mutations are refused, placement and advertised availability are disabled, and loads are retried after repair. Writes fsync before and after the rename. GET /resources reports allocatable_slots and quarantined_slots, and GPU admission gates on the allocatable count. GPU.md documents the store semantics, draining the parent GPU, the SR-IOV recovery cycle, and clearing quarantine state.
A failed state write previously rolled memory back and left the store reporting healthy, so a VF whose threshold-crossing failure could not be persisted stayed allocatable. Latch write failures and refuse placement until a later write succeeds; re-reported markers retry the write. Also make acknowledged reports crash-durable (fsync the parent when the state dir is first created, treat directory sync failures as persist failures instead of logging success), and re-evaluate persisted tallies against the configured threshold at load and on threshold changes so a lowered gpu.vf_quarantine_threshold applies to existing failures.
Fold the below-threshold placement assertion into TestVGPUAvailability and the repaired-state recovery assertion into TestVGPUAvailabilityFailsWhenStoreUnavailable, exercising both through the public API. Drop TestReportVFInitFailureRespectsConfiguredThreshold and TestCheckedAddressesFailsClosedOnUnloadedState, whose remaining coverage is subsumed by the threshold re-evaluation and invalid-record tests.
94f8496 to
5a8fcae
Compare
Summary
Bottom half of the wedged-VF work (detection lands separately on top, in #435). This PR adds the quarantine machinery: a persisted VF health store plus the placement, admission, and API changes that act on it. Nothing in this PR reports failures automatically yet — the store is fed by the guest-side detection in #435, and by operators editing the state file per the runbook.
<data-dir>/gpu/vf-health.jsonper instance assignment; a VF is quarantined once failures accumulate fromgpu.vf_quarantine_thresholddistinct assignments (default 2). An exact-assignment success report clears the match and older tallies, and rescinds that assignment's quarantine.GET /resourcesreportsallocatable_slotsandquarantined_slots; GPU admission gates on the allocatable count.Safety and failure handling
lib/devices/GPU.mddocuments the store semantics, draining the parent GPU, the SR-IOV recovery cycle, and clearing quarantine state.Testing
Passed:
go test -race ./lib/devices ./lib/paths ./lib/resources ./cmd/api/config go vet ./lib/devices ./lib/resources ./lib/paths ./cmd/api/config ./cmd/api/apiThe full
lib/instancessuite was also attempted;TestCreateInstanceWithNetworkfails in this environment on both this branch and its base (it boots a real VM image), so it is environmental, not introduced here. The stacked detection PR's tree is byte-identical to the previously reviewed and tested head of #435.Note
Medium Risk
Changes vGPU placement, admission, and persisted state with fail-closed behavior; misconfigured or corrupted vf-health.json can block new vGPU placements until repaired, but running VMs are not torn down.
Overview
Adds vGPU VF quarantine for wedged virtual functions: guest driver init failures are tallied in persisted
vf-health.jsonunder the data dir, and VFs are quarantined after failures fromgpu.vf_quarantine_thresholddistinct instance assignments (default 2, configurable and validated at startup).Placement and capacity: Vendor-VFIO selection skips quarantined VFs, deprioritizes parent GPUs that have quarantined VFs, and randomizes among equivalent free VFs on the chosen card. Profile availability and
VGPUAvailabilityexclude quarantined VFs. Corrupt or unwritable health state fails closed (no placement / no advertised allocatable capacity until load or persist succeeds).API and admission:
GET /resourcesGPU status addsallocatable_slotsandquarantined_slots; GPU reservation/validation gates on allocatable slots and returns explicit errors when VF health is unavailable.ReportVFInitFailure/ReportVFInitSuccessAPIs exist for future guest detection (#435) and manual operation; this PR does not wire automatic failure reporting.Docs (
GPU.md,config.example.yaml) cover store semantics, draining, and SR-IOV recovery. Extensive unit tests cover the health store and placement behavior.Reviewed by Cursor Bugbot for commit 5a8fcae. Bugbot is set up for automated code reviews on this repo. Configure here.