e2e: kubernetes suites fail cleanup because one node's ZFS pool stays 450-830 MiB short of its baseline

[lexfrei]

Since 2026-10-01, the kubernetes e2e suites fail their cleanup in the PR lane because one node's ZFS pool does not come back to its pre-suite level. The suite's own checks all pass; it goes red in cozy_cleanup, which compares each node with the baseline within a 512 MiB tolerance (cozy_linstor_pools_at_baseline in hack/e2e-chainsaw/_lib/run-kubernetes.sh). Each time, 450-830 MiB stays used on one or two nodes and is never freed.

Runs, in KiB (baseline → after cleanup):

• 36844521765 (fix(e2e): bound the dataplane capture collector by wall clock #4640), kubernetes-latest: srv3 74348197 → 73715123, 633074 short. srv1 and srv2 came back exactly, and kubernetes-previous in the same run came back.
• 36854302536 (ci(workflows): pin flux, yq and helm-unittest installs #4645), kubernetes-latest: srv1 832247 short, srv3 463379 short.
• 36831242577 (renovate/indirect-go-modules, Go modules only): both kubernetes suites, srv1 541460 short, then srv3 793677 short. srv1's baseline for the second suite equals what the first one left, so the space is not released later.

Passing runs for comparison: 36785869560 on 2026-09-30 (under 1 MiB of drift) and 36845137883 on 2026-10-01 09:48 UTC. None of the failing PRs touches LINSTOR, the tenant teardown or run-kubernetes.sh, and the renovate branch failed before the Talos bump (#4598) merged.

The likely cause, not proven: a CDI scratch volume whose first CreateVolume failed leaves a partly created zvol behind. In 36844521765 the scratch PVC pvc-f1883e08 first failed with ResourceExhausted ... Not enough available nodes at 12:12:38 UTC, was then provisioned and attached on srv3, later logged VolumeFailedDelete ... still attached to node srv3, and is absent from the final LINSTOR resource list. srv3 is the node that came back short. For the other runs the events had already expired when cozyreport was collected.

What is missing to prove it: cozyreport has no zfs list of the pools, so the dataset holding the space can't be named, and a snapshot or ZFS metadata hasn't been ruled out.

Fix shape: first, on this failure print zfs list -t all -o name,used,refer,origin for every data-srvN pool together with the LINSTOR resources, so the next red names the dataset. Then fix the leak where it is, not by widening the 512 MiB tolerance, which would hide a real leak. #4651 changes only the replicated lane and does not affect this.

[lexfrei]

The space is freed. What stays short is LINSTOR's cached figure, so this is not a leaked zvol.

For a node whose pools are all thick ZFS, linstor sp l does not ask the satellite for free space. The controller fetches it live only for nodes with thin pools and otherwise returns the value from the satellite's last report. At LINSTOR v1.33.3 (our pinned version) that is FreeCapacityFetcherProto.assembleRequests filtering on hasThinPools, with CtrlStorPoolListApiCallHandler falling back to getFreeCapacityLastUpdated. The e2e pool data is plain ZFS, so the 300s wait in cleanup keeps reading the same frozen number.

The job logs agree with that. The residue shows up only on the nodes that hosted the suite's worker disks, and the other nodes match to the KiB. When the next suite records its baseline about six minutes later, it equals the failed cleanup's "after" value to the KiB, which is the frozen value and not space that is still held. At the end of each job, used space on every node equals 1.0156 × the sum of the LINSTOR volume sizes plus a fixed slop, and every LINSTOR resource maps to a Bound PVC. My guess for why the cached value is short is that the satellite reports right after zfs destroy, while the written blocks are still being freed in the background. I haven't measured zpool get freeing at that moment.

Something did change on 2026-10-01. In a sample of in-tree runs from 2026-09-17 on, the check passed in 37 of 38 jobs with 0 to 2 MiB of drift. The one failure, run 36295991339 on 2026-09-27, was a real 21 GiB scratch volume left on srv3. On main base 0592d0e none of 7 runs had a residue, and on base 031ded7 or newer 12 of 14 did, the green ones included. Main has only two commits between those bases. Neither looks like it can cause this, so the trigger may be in the environment rather than in the code. The cache is why the check reacts to it at all.

A byte tolerance can't be tuned around this. The residue has reached 910 MiB, and a tolerance above 1 GiB would miss a leaked 1 Gi PVC. #4651 now replaces the byte comparison in both lanes with the set of ZFS datasets per node, read with zfs list in each satellite. A destroyed dataset disappears from the list when zfs destroy returns, so a leak shows up by name and background freeing can't fail the check. It closes this issue.