fix(reachability): announce a target that is never called (finding B) - #60
Merged
Merged
Conversation
TargetPass instruments call sites whose callee matches the requested name. When the program never calls that function, nothing is instrumented and the reachability property holds trivially -- and nothing in the output says so. Zero call sites is not an error. A program that genuinely never calls reach_error IS safe with respect to reach_error, and TRUE is correct there. The defect is that such a TRUE is indistinguishable from a TRUE earned by exploring the program. So the fix is visibility, not a change of verdict: the pass scans the module once and, when the target has no call sites, says what the verdict is worth. Measured on the CASTLE v6 baseline, which is what moves this from a suspicion to a defect with a number. 98 of 217 runs used `--target-function-name main` -- a program does not call its own entry point: TRUE 57, TIMEOUT 22, UNKNOWN 10, ERROR 9, FALSE-* 0 Not one FALSE, while the other modes produce 49 across the same corpus. 59 of those 98 programs were known to be vulnerable and none was detected. Seventeen were scored: 5 FN, and 12 TN -- credit for answers an oracle that can only say "correct" could not have failed to give. Excluding the mode, recall goes from 74.0% to 79.4%; precision is unchanged at 98.2%, since the mode yields neither TP nor FP. Correcting my own earlier reading: the 61 N/A rows are N/A because the harness declares those CWEs out of scope, not because of finding B. B's real cost is the 17 scored cases. Still open, deliberately: the harness maps 13 CWEs onto this mode as an escape default. Fixing that changes baseline scoring, so it waits for the v6 run to finish rather than moving the criterion mid-run. Verified end to end against a full build, not just the pass in isolation: test_target_coverage.sh 4/4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finding B, measured
Finding B was logged in August as a structural suspicion:
TargetPassinstruments call sites whose callee matches--target-function-name, and a program does not call its ownmain, so nothing gets instrumented. The CASTLE v6 baseline turns the suspicion into a number.98 of 217 runs (45%) went through
--target-function --target-function-name main:Not one FALSE. That is not scarcity — across the same 217 runs the other modes produce 49 FALSE-* verdicts. This mode is the only one that never accuses anything.
59 of those 98 programs were known to be vulnerable. Zero were detected.
The scoring damage has two sides, and the second is the uncomfortable one:
Excluding the mode entirely: recall 74.0% → 79.4%, precision unchanged at 98.2% (the mode yields neither TP nor FP). So B does not inflate precision — it depresses recall and inflates TN.
Correcting an earlier reading of mine: the 61 N/A rows are N/A because the harness declares those CWEs out of scope, not because of finding B. B's real cost is the 17 scored cases.
What this PR changes
Zero call sites is not an error. If a program genuinely never calls
reach_error, then "the error is unreachable" is the right answer and TRUE is correct. The defect is that such a TRUE is indistinguishable in the output from a TRUE earned by exploring the program.So the fix is visibility, not verdicts.
TargetPassscans the module once and, when the target has no call sites, says what the verdict is worth there:Reported once per module rather than once per function — this is a function pass with no module-entry hook, and
optruns once per invocation.Tests
tests/integration/test_target_coverage.sh, wired into the regression job. Four assertions, and it asserts both directions — a warning that always fires is as useless as one that never does:Verified against a full build and install, not the pass in isolation.
Deliberately not in this PR
The CASTLE harness maps 13 CWEs onto this mode as an escape default (
tests/castle/run_castle_evaluation.sh:34-47). That is the other half of finding B, and fixing it changes baseline scoring — so it waits for the v6 run to finish. Changing the scoring criterion mid-run is exactly what makes two measurements incomparable.Docs
docs/reports/2026-08-12-castle-juliet-findings.md— new §2.1 with the measurement.docs/TESTCOMP-CHECKLIST.md— refreshed: K moved to fixed, B promoted to High with its number, the three image defects from fix(docker): build Clam with clang-16, and validate Dockerfile.dev on PRs #58/chore(docker): pin Clam and its three dependencies to SHAs #59 recorded, v6 status updated. It also now records why K measured as zero effect on CASTLE: finding B masks it, since a mode that never records a violation gives K nothing to rescue.