just validate-references performs zero checks, so the verbatim-snippet guarantee is untested

[realmarcin]

Found in the review of #465, and it explains how eight truncated snippets survived.

snippet is meant to be a verbatim quote from the cited paper — the evidence model rests on it. snippet carries implements: linkml:excerpt in the schema, and conf/reference_validator.yaml points at a populated references_cache/ (715 entries). But the validator runs no checks at all:

$ just validate-references kb/communities/Geobacter_Clostridium_Interspecies_Electron_Transfer_Coculture.yaml
Validation Summary:
  Total checks: 0
  All validations passed!

Same for every record tried, and identical on main. So it passes vacuously — a green run says nothing was checked, not that anything is right.

That is how a snippet reading "...couple the electron balance with o" survived, and seven more like it across five records (all repaired in #465). Nothing else covers this: validate-scalars (#398) detects a different defect — a plain scalar swallowed by a # comment — and correctly reports 0 for these, which are well-formed scalars that merely stop early.

CLAUDE.md says validate-references does "snippet validation against abstracts", and the recipe comment says it "can verify snippets taken from a paper's Methods/Results". Neither is currently true.

Worth determining whether this is a configuration problem (the validator not finding the slot or the cache), a version change in linkml-reference-validator, or something the qc-references path exercises differently. Note #417 took validate-references-all out of qc because it failed on main — worth checking whether that path does real work while the single-file one does not.

#465 added tests/test_snippets_are_not_truncated.py, which does the verbatim comparison directly against references_cache/ and catches the mid-word class. That is narrower than what this validator is supposed to do (it says nothing about paraphrase — #347) but it is currently the only thing checking snippets against sources at all.

[realmarcin]

Closing as substantially incorrect, with the correction in #475 (6bb437e).

The headline claim — that validate-references performs zero checks and passes vacuously — does not hold. Total checks is printed as len(all_results) in the upstream CLI (linkml_reference_validator/cli/validate.py:313), and all_results holds validation issues. Total checks: 0 means no problems found. A clean record prints 0 and always will.

Planting a snippet that appears in no publication into taxonomy[0].evidence[0]:

[ERROR] Text part not found as substring: 'ZZQQ this sentence appears in no publication whatsoever XKCD'
  Location: taxonomy[0].evidence[0].snippet
Total checks: 1 / Issues found: 1

The justfile already documented this from #257, in four lines directly above the recipe. I filed this issue anyway, so the fix in #475 is to stop relying on prose: tests/test_reference_validator_actually_validates.py now fails if the validator ever accepts fabricated text, and a second test asserts a real curated snippet is still accepted — a checker that rejected everything would pass the canary alone.

Two things this issue got right, for different reasons than it gave:

1. The eight truncated snippets in #262's DIET re-scoping missed one occurrence: same snippet still cited as SUPPORT #295/Apply #262's DIET judgement to the occurrence it missed (#295) #465 did survive this validator — but because a quote cut mid-word is a substring of the source, and substring matching is exactly what the tool promises. Not vacuity. tests/test_snippets_are_not_truncated.py is the right cover for that class.

2. Something genuinely is unchecked, just not what was reported: SupportingReference, the range of Discussion.evidence, carries no implements: annotations, so 11 snippets across 8 records are never validated. Filed as SupportingReference snippets are never validated: Discussion.evidence has no implements: annotations (cross-repo, mech_shared.yaml) #476 — it needs a coordinated cross-repo change because mech_shared.yaml is vendored byte-identical and sha-pinned across the four Mech repos.

That second finding is what made this issue worth working, and it is also what misled me at first: my initial plant landed in discussions[0].evidence[2] — a SupportingReference — and appeared to confirm the vacuity claim. Two different findings producing identical output.