Skip to content

Re-audit degenerate-container UB, and put the class under CI - #178

Merged
ms609 merged 2 commits into
cpp-searchfrom
feature/ub-class-audit
Aug 8, 2026
Merged

Re-audit degenerate-container UB, and put the class under CI#178
ms609 merged 2 commits into
cpp-searchfrom
feature/ub-class-audit

Conversation

@ms609

@ms609 ms609 commented Aug 7, 2026

Copy link
Copy Markdown

Fixes #177.

The July sign-off on degenerate-container UB was wrong twice in a week
(#124, #151), both times for the same reason: it argued reachability
transitively — "callers cannot produce zero words" — instead of checking,
and one of its false claims went into the memory archive, where it then hid
#124 from a later reader for a month. So this redoes the audit, and, more
usefully, stops the class depending on a human pass at all.

Why one detector was never going to be enough

Sub-class Example Seen by Blind to it
Null pointer to a nonnull parameter memcpy(v.data(), w.data(), 0), v empty UBSan nonnull-attribute (gcc-ASAN) _GLIBCXX_ASSERTIONS, plain builds
Out-of-range address formation &v[0], v.back() on empty v — no load or store _GLIBCXX_ASSERTIONS (glibcxx-assertions, from #60) ASan, which watches accesses, not address arithmetic

Neither leg sees the other's sub-class, so "the sanitizers are clean" says
something only once both have executed the shape. Coverage here is bounded
by which inputs reach the detectors — never by reading code. That is the
finding, and it is why the deliverable is a test file rather than a report.

Static pass — all 69 memcpy/memmove/memset sites

Each classified as guarded, or provably non-empty at the site; a proof
that appeals to what callers can produce does not count.

Group Sites Verdict
MaddisonSlatkin.cpp, ts_data.cpp _pad 19 Fixed-size C arrays; sizeof(data) cannot be zero
ts_tbr.cpp:930 1 Two scalars
TbrSnapshot::save/restore 12 Guarded — #134
ts_bench_tbr_phases 12 Guarded at entry — #174
ts_tree.cpp load_tip_states, save_node_state 8 Guarded on tip_bytes > 0 / total_words == 0
ts_tree.cpp restore_prealloc_undo 5 Loop is while (u.count > 0); save_node_state returns before incrementing count
ts_prune_reinsert, ts_sector, ts_collapsed 3 Guarded on tw > 0 / early return
ts_tbr.cpp L3b, vroot, NA clip 8 Under l3b_active / use_directional (both require total_words > 0) or has_na (implies ≥1 block)
ts_splits.cpp 2 wps >= 1; emit-loop filters identical to the count loop's

No unguarded site. The siblings .front() / .back() / .data() + i were
swept too (45 + 11); every .back() is inside a while (!stack.empty()) or
on postorder, which is never empty.

Dynamic pass — 1704 runs under -D_GLIBCXX_ASSERTIONS

Battery Runs Axis
1 663 13 degenerate shapes × ~30 entry points × {EW, IW}
2 382 Hierarchy/HSJ/XFORM, constraints, profile parsimony, resampling API, 24- and 40-tip trees, 12-state characters
3 659 37 TS_* environment knobs, plus the ≥150-tip regime that activates L3b

Zero aborts. Battery 3 is the one that mattered: l3b_active needs
n_tip >= 150 unless TS_L3B_INCREMENTAL is set, so eight ts_tbr.cpp
sites had never been executed by any test at any point.

Positive control, because "no aborts" is uninterpretable without proof
the harness can abort: reverting the #151 guard and rebuilding gives
stl_vector.h:1130: Assertion '__n < this->size()' failed, rc=127, on the
first degenerate call; restoring it returns the run to green.

What lands

tests/testthat/test-ts-degenerate-shapes.R — Tier 2, 329 expectations,
~5 s. Its expectations are contract checks; its job is to put these shapes
in front of both sanitizer legs on every dispatch. It aborts against a
build with the #151 guard removed
, so it is not tautological.

It forces L3b with TS_L3B_INCREMENTAL on a 12-tip tree instead of paying
for 150. l3b_active needs more than that knob — also a null sector mask,
no tabu list, no pool collection — so rather than assume, I checked: under
TS_L3B_STATS=1 the forced call reports patch_clips=36. The zero-word
datasets in the same test are the other side of that guard, where L3b is
correctly inert.

One code fix. A 0 x 2 startEdge matrix satisfied every existing
shape check, left flat empty, and reached flat.data() + n_edge
pointer arithmetic on a possibly-null pointer, undefined before C++20 and
reported by neither detector. Without the new guard the existing
test-ts-driven.R case fails with Each startEdge matrix must describe a binary tree, i.e. it demonstrably reached the site.

What this does NOT establish

  • _GLIBCXX_ASSERTIONS hardens libstdc++ containers only. Raw arrays, and
    raw pointers derived from a container (const uint64_t* bits = &v[i];
    then bits[w]), are unchecked — those reads are ASan's job.
  • Rcpp vector indexing is hardened by neither flag.
  • The dynamic pass is bounded by the shapes I imagined. Every historical
    instance was a zero-Fitch-word dataset, so that axis is now covered hard
    and others less so.
  • Battery 3's knobs were set; that each switched on the path it names was
    not verified individually. Only L3b was load-bearing, and only L3b was
    confirmed.

Write-up: dev/red-team/reviews/container-ub-class-audit/. A new
dataset-taking entry point should join test-ts-degenerate-shapes.R in the
commit that introduces it.

CI

agent-check 31210301540
— green, all three jobs. The libstdc++ hardened assertions leg reports 34
compiler invocations carrying the flag and zero assertion failures.

gcc-ASAN 31210334799
0 runtime error, 0 nonnull reports, which is the result this PR
turns on. The tests job is nonetheless red, for a reason that predates
this branch: test-Concordance.R:433 calls tools::Rd_db("TreeSearch"),
and the ASan container installs without a help database, so it errors with
"installed help of package 'TreeSearch' is corrupt". The identical failure
is on run 31193721419
(feature/consistency-fixes, three hours before this branch existed).
Filed separately; nothing here touches Rd or roxygen.

Both legs demonstrably executed the new file rather than skipping it:
ts-degenerate-shapes reports 329 expectations in each, matching the
local count exactly, and ts-driven reports 185 (184 + the one new case).


Attribution: ms609-agent is suspended, so gh writes — this PR, #177,
and its comments — land under ms609 rather than the agent account.

ms609-agent added 2 commits August 7, 2026 20:10
The July sign-off on this class was wrong twice in a week (#124, #151),
both times because it argued reachability transitively rather than
checking. Redo it properly, and — the part that lasts — stop relying on
a human pass at all.

The class needs two detectors that are blind to each other: UBSan's
nonnull check sees a null `.data()` reaching `memcpy`, and hardened
libstdc++ sees `&v[0]` on an empty vector, which forms an out-of-range
address without ever loading from it. Both legs now exist in CI, so
coverage is bounded by which *inputs* reach them — not by code reading.

Static pass over all 69 memcpy/memmove/memset sites plus the .front() /
.back() / .data()+i siblings: every one is guarded or provably non-empty
at the site. Dynamic pass: 1704 runs under a local -D_GLIBCXX_ASSERTIONS
build, over degenerate shapes x entry points x weighting modes, the 37
TS_* alternative kernels, and the >=150-tip L3b regime that no test had
ever entered. No aborts — and a positive control proves that statement
means something: reverting the #151 guard aborts on the first call.

Adds tests/testthat/test-ts-degenerate-shapes.R (Tier 2, ~5 s), whose
job is to put these shapes in front of both sanitizer legs on every
dispatch. It aborts against a build with the #151 guard removed, so it
is not tautological.

One code fix: a 0 x 2 `startEdge` matrix passed every existing shape
check and reached `flat.data() + n_edge` on an empty vector — undefined
before C++20, and reported by neither detector.

Fixes #177
`l3b_active` needs more than the TS_L3B_INCREMENTAL knob -- also a null
sector mask, no tabu list and no pool collection -- so "the knob buys the
L3b sites" was a reachability claim of exactly the kind the July audit got
wrong. Checked it: TS_L3B_STATS=1 reports patch_clips=36 on the forced
12-tip call, so the path does engage. Say so where the claim is made.

Also note that the zero-word datasets in the same test are the other side
of that guard -- L3b is correctly inert for them -- and hedge the audit
note's env-knob row, where per-knob path engagement was not verified.
@ms609
ms609 merged commit 5e6396b into cpp-search Aug 8, 2026
11 of 12 checks passed
@ms609
ms609 deleted the feature/ub-class-audit branch August 8, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-audit degenerate-container UB across the codebase, and make the class continuously observable

1 participant