capture_vm_ssh_host_keys (#324) is only called from check_vm (lnvps_api/src/worker.rs:994), which itself only runs via WorkJob::CheckVm — dispatched on start/stop/restart/reinstall (api/routes.rs:1411,1431,1456,1525) and admin manual payment completion (admin/vms.rs:1325). None of those is a recurring background pass.
The actual periodic fleet check is WorkJob::CheckVms (scheduled every 30s, bin/api.rs:318) → check_vms() → check_vms_on_host() (worker.rs:1180) — bulk state fetch, handle_vm_state, reconcile_vm_dns, per VM. No call to capture_vm_ssh_host_keys in that path. #324's own PR description says capture "hangs off the periodic VM check, next to the DNS reconcile" — it is next to a DNS reconcile call, just the one inside check_vm (the reactive, one-off path), not the one inside check_vms_on_host (the actual 30s sweep), which has an identical reconcile_vm_dns(vm) line already.
Net effect: host keys only populate for a VM a customer happens to start/stop/restart/reinstall, or that an admin manually completes payment on. Any VM nobody touches keeps host_ssh_keys empty indefinitely. This is what Kieran is seeing.
Fix
Add self.capture_vm_ssh_host_keys(vm).await; to check_vms_on_host's per-VM loop (worker.rs:~1196, right after the existing reconcile_vm_dns(vm).await;), mirroring check_vm's own ordering.
No separate backfill job needed
Once wired into check_vms_on_host, every non-deleted VM is already visited there every 30s (rate-limited globally via the last-check timestamp, not per-VM). capture_vm_ssh_host_keys already no-ops on a VM whose capture is complete and self-throttles retries to hourly per VM (HOST_KEY_SCAN_RETRY_SECS). The whole fleet, including every VM that predates #324, self-heals on the next sweep after this one-line fix — a bespoke backfill script would duplicate logic that already exists and is already safe to run unconditionally.
capture_vm_ssh_host_keys(#324) is only called fromcheck_vm(lnvps_api/src/worker.rs:994), which itself only runs viaWorkJob::CheckVm— dispatched on start/stop/restart/reinstall (api/routes.rs:1411,1431,1456,1525) and admin manual payment completion (admin/vms.rs:1325). None of those is a recurring background pass.The actual periodic fleet check is
WorkJob::CheckVms(scheduled every 30s,bin/api.rs:318) →check_vms()→check_vms_on_host()(worker.rs:1180) — bulk state fetch,handle_vm_state,reconcile_vm_dns, per VM. No call tocapture_vm_ssh_host_keysin that path. #324's own PR description says capture "hangs off the periodic VM check, next to the DNS reconcile" — it is next to a DNS reconcile call, just the one insidecheck_vm(the reactive, one-off path), not the one insidecheck_vms_on_host(the actual 30s sweep), which has an identicalreconcile_vm_dns(vm)line already.Net effect: host keys only populate for a VM a customer happens to start/stop/restart/reinstall, or that an admin manually completes payment on. Any VM nobody touches keeps
host_ssh_keysempty indefinitely. This is what Kieran is seeing.Fix
Add
self.capture_vm_ssh_host_keys(vm).await;tocheck_vms_on_host's per-VM loop (worker.rs:~1196, right after the existingreconcile_vm_dns(vm).await;), mirroringcheck_vm's own ordering.No separate backfill job needed
Once wired into
check_vms_on_host, every non-deleted VM is already visited there every 30s (rate-limited globally via the last-check timestamp, not per-VM).capture_vm_ssh_host_keysalready no-ops on a VM whose capture is complete and self-throttles retries to hourly per VM (HOST_KEY_SCAN_RETRY_SECS). The whole fleet, including every VM that predates #324, self-heals on the next sweep after this one-line fix — a bespoke backfill script would duplicate logic that already exists and is already safe to run unconditionally.