CI baseline for the capture path: real sink chain, multipart on MinIO, backend compile, undefined-name lint

[zaoxing]

Stacked on #143. Merge #143 first; GitHub then retargets this PR to main.

Changes

(a) The live job proves the real sink-to-reader chain. It now also builds build/_dmi_native_sink. The new tests/test_native_capture_chain_live.py drives the real NativePackSink (through its attach/submit_envelope entry points, with no CUDA and no conformance driver) → spool → in-process CaptureStorageService → ClickHouse plus object store → NativeCaptureReader, with byte-equal payloads. It has three cases:

• f16, bf16, f32 and i64 rows in 3 packs, against the in-repo fake S3;
• a 68 MiB single-pack multipart upload (5 parts) against the fake S3. The test asserts the part sizes the client sent, because the fake enforces no part-size rules;

(b) A new native-backend-compile job, which is the first time CI compiles _native_backend at all. On ubuntu-24.04 it:

• frees disk;
• initialises only the clickhouse-cpp submodule;
• installs a minimal CUDA 13.0 from NVIDIA's apt repository (about 2.7 GB; the package list comes from the build's own dependency files);
• installs torch 2.14 cu130 and builds clickhouse-cpp as install.md does;
• runs make -C native all against the libcuda stub;
• imports the backend;
• builds the six tests/native/ring binaries and runs the two that need no device.

A step applies #138's C++20 flags by pattern until #138 merges. After that it matches nothing and asserts all three lines read c++20. Delete that step once #138 lands. A container image was rejected: the 3.7 GiB CUDA devel image plus about 4.4 GB of torch wheels doesn't reliably fit a hosted runner.

(c) Undefined-name lint. The cpu job runs ruff check --select F821 src/dmi. The tree had 14 existing violations, handled visibly in their own commit (f48becd), which you can drop if you prefer another fix:

• src/dmi/config.py (2): MonitoringConfig's string annotations "NativeSinkConfig" and "NativeCaptureStorageConfig" were never imported. They are now imported under TYPE_CHECKING. Latent, not fixed here: typing.get_type_hints(MonitoringConfig) still raises NameError; B2's move of NativeSinkConfig into live code is the natural fix.
• src/dmi/hooks/specs.py (12, on 10 lines): the HOOK_TYPE_* names are injected by a globals() loop the linter can't follow, so these are false positives. They're suppressed per line, not per file, so the rest of the module stays checked. The alternative, declaring the constants statically, is a maintainer call.

Evidence (local)

• Chain tests: 2 passed, 0 skipped (after MinIO was removed; see the update below).
• The tests catch a real break. With the client's multipart chunk forced to 4 MiB: the fake-S3 case fails on part count (18 vs 5).
• Full live glob, as CI runs it: 211 passed, 1 failed. The failure is a role test that needs a CREATE USER grant this local server lacks. The live skip gate reports 0 skipped.
• CPU tier: 2213 passed, 1 skipped (no CUDA device); the cpu skip gate is clean.
• Backend build steps, rehearsed locally (C++20, CUDA 13.0, sm_89, libcuda stub): builds, links without libcuda, and imports with devices hidden. The stock C++17 flags fail with ATen's #error, as expected.
• Lint: 14 violations at the base; clean at the head; an injected undefined name fails the step with a GitHub annotation.
• An independent review re-verified all of the above and found no defects.

Not verifiable before CI runs: the compile job on the hosted runner (apt install, disk headroom, gcc 13, runtime under the 60-minute timeout), and pip install ruff there. Making these required checks on main is a branch-protection change I haven't made.

Since the independent review (2026-09-24)

An independent review found one major, three minors and some nits; all are fixed (eb79cfd..e68fea2), and one CI break was fixed after (a2033b3):

• Major: the chain test now compares every catalog metadata field (all 21) against what was submitted. A mutant sink that stored step_number + 7 used to pass the chain test and all of CI; it now fails it.
• The multipart part count is deduplicated by part number, so one transient HTTP retry cannot fail it.
• The compile job now also builds _dmi_native_sink beside the backend and asserts the sink binds the backend's RecordSink (not stand-ins), through dmi's loaders and via a bare import. A pytest check alone stayed green with the binding broken, which is why the assertions are there.
• The temporary C++20 step emits a ::warning:: once Build the torch-including native targets as C++20 #138 has landed and it has nothing left to change.
• Comments no longer claim numbers or releases that were not observed.

MinIO removed: MinIO later withdrew its public images, and nothing in DMI runs against MinIO (its object store is Garage), so the MinIO step and test case were dropped (latest commit). The fake-S3 multipart case, which asserts the part sizes the client sends, remains; checks against a real store belong to the Garage suites.

Evidence before MinIO was removed (CI at a2033b3): live suite 219 passed, 0 skipped; native-backend-compile green, sink binding "1 passed" and 21 ring tests passed; CPU job 2234 passed.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.