Make "does validate-references check anything" a test, not a note (#466)

[realmarcin]

Addresses #466 by correcting its premise and closing the gap that diagnosing it actually revealed.

#466's core claim is wrong

The issue read this as a vacuous pass:

Validation Summary:
  Total checks: 0
  All validations passed!

Total checks is printed as len(all_results) in the upstream CLI (linkml_reference_validator/cli/validate.py:313), and all_results holds validation issues. So Total checks: 0 means no problems found. A clean record prints 0 and always will.

Demonstrated by planting a snippet that appears in no publication into taxonomy[0].evidence[0]:

Validation Issues (1):
  [ERROR] Text part not found as substring: 'ZZQQ this sentence appears in no publication whatsoever XKCD'
    Location: taxonomy[0].evidence[0].snippet
    Reference: PMID:28287150
Total checks: 1 / Issues found: 1

EvidenceItem snippets are genuinely validated against references_cache/.

Worth noting for process: the justfile already said this, from #257, in four lines immediately above the recipe — "It IS validating (a fabricated snippet fails it); only the label is wrong." #466 was filed anyway. That is the argument for the change below: prose next to the code was not enough.

A detail that cost me an hour, recorded so it doesn't cost the next person one

My first plant did appear to confirm #466 — gibberish went in, validator passed. The snippet had landed in discussions[0].evidence[2], which is a SupportingReference, not an EvidenceItem. Two different findings wearing the same output.

What this changes

tests/test_reference_validator_actually_validates.py — five tests, no subprocess and no network, reading the same references_cache/ the recipe does:

• the canary: fabricated text must be rejected. If this ever passes, every green validate-references run is meaningless.
• the converse: a real curated snippet must still be accepted — a checker that rejects everything passes the canary alone and is equally useless.
• the implements: annotations that make the check reachable at all.
• the SupportingReference gap, pinned (below).
• the size of that gap, so it cannot be deferred as hypothetical.

Plus a justfile note pointing the #257 prose at the now-mechanical proof.

The real finding: SupportingReference is never checked

Discussion.evidence ranges on SupportingReference. Unlike EvidenceItem, its snippet and reference carry no implements:, so the plugin's field detection cannot see them:

class	_find_excerpt_fields	_find_reference_fields
EvidenceItem	['snippet']	['reference']
SupportingReference	[]	[]

11 snippets across 8 records are unvalidated. The module's own description claims the opposite — that it "carries a verbatim snippet so the same anti-hallucination snippet-vs-cached-abstract check the Mechs already run can validate it."

Deliberately not fixed here. It lives in mech_shared.yaml, which is vendored byte-identical and sha-pinned across the Mech repos (scripts/check_vendored_sync.sh:33), and the file says "Do not edit one copy in isolation — change it once and re-vendor + re-pin everywhere." Editing this copy alone fails the vendored-sync gate. Filed as a follow-up.

The gap is pinned by a test that fails when it is closed, so it cannot be closed silently — whoever fixes it gets a red test pointing at the docstring explaining what to update.

What stays true about the truncation class

A snippet cut mid-word still passes, because a cut quote is a substring and substring matching is what the tool promises. That is why the eight in #295/#465 survived — not vacuity. tests/test_snippets_are_not_truncated.py covers that class; this PR does not change it.

No behavioural change: tests and comments only. ruff, black clean.

🤖 Generated with Claude Code