Apply #262's DIET judgement to the occurrence it missed (#295) - #465
Merged
Conversation
The record cites the same PMID:28287150 snippet three times, not twice as the issue says. #262 re-scoped two to PARTIAL on the grounds that it is the paper's generic background about Geobacter species in prior literature, not a finding about this coculture. The third stayed SUPPORT, so the record simultaneously held that the snippet establishes nothing here and that it fully supports an interaction. Worse than the issue recorded: that third occurrence also had no explanation and a snippet truncated mid-word at "with o". #262's note on the first occurrence says "Snippet also completed here, having previously been truncated mid-word" - the same truncation, fixed in one place and left in another. Re-scoped to PARTIAL with the same reasoning, snippet completed, explanation added. The interaction is unaffected either way: its other two items, the metabolic-shift quantification and the electron-uptake measurement, are what carry it - which is what the issue predicted. Writing that explanation reintroduced #400's defect, in the file I was fixing: an unquoted "#262" in a plain multi-line scalar starts a YAML comment, so everything after it was swallowed and the file stopped parsing. `just validate` caught it. Quoted, and swept the KB for the same pattern - three continuation lines contain " #", all inside quoted scalars, and validate-scalars reports 0 truncated across 318 files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Network integrity findingsWarnings only — a member with no interaction yet, or a participant matched by ontology id rather than by name, or one on a community-level interaction that resolves to no member. Reported, but does not fail the build. The full report is attached to the workflow run as an artifact. |
I claimed the KB was clean on the basis that validate-scalars reports 0 truncated across 318 files. That check structurally cannot see this defect - it flags a plain scalar swallowed by a "#" comment, and these are well-formed scalars that merely stop early. It reports 0 correctly and says nothing. The reviewer's sweep - resolve each snippet against its cached source, flag any that matches verbatim but is followed in the source by a letter - found seven more, in five records: "...ammonia-oxidizing bac" -> bacteria (x2) "...and Stenotrop" -> Stenotrophomonas (x5) "...elemental sulf" -> sulfur "...were potential" -> potentially "...propionate usi" -> using protons as the electron acceptor "...were upregu" -> upregulated in D. vulgaris "...base of the Thermo" -> Thermoplasmatales within the Euryarchaeota All completed verbatim from references_cache/, and the sweep is now a test so the class cannot recur silently. Digits following a snippet are excluded - those are citation markers the cached markdown ran together with the preceding word - and one letter case is allowlisted with its reason: "parvus" + "cocultivated" is a missing space in the cache, not a cut quote. Also stopped the new explanation vouching for a sibling item that does not do what its own explanation claims: that snippet is the paper's study-scope sentence and quantifies nothing, despite an explanation saying it quantifies the shift toward 1,3-propanediol. Same defect class as this one; noted rather than fixed, since it needs the curator's judgement. Filed #466: just validate-references performs zero checks and passes vacuously, which is how all eight of these survived. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
Review round 1 — my "the KB is clean" claim was unsupported, and wrongI justified it with The reviewer's sweep — resolve each snippet against its cached source, flag any that matches verbatim but is followed in the source by a letter — found seven more, across five records:
All completed verbatim from Two refinements the sweep needed: digits following a snippet are excluded (citation markers the cached markdown ran together with the preceding word — Also fixedMy new explanation vouched for a sibling item that doesn't do what its own explanation claims — that snippet is the paper's study-scope sentence and quantifies nothing, despite an explanation saying it quantifies the shift toward 1,3-propanediol. Same defect class, six lines up. Noted rather than silently fixed, since re-scoping it is the curator's judgement. Why none of this was caught — #466
Identical on Verified clean by the reviewThe snippet completion is verbatim against
|
realmarcin
added a commit
that referenced
this pull request
Aug 7, 2026
…) (#475) #466 reported the reference validator as performing zero checks, reading Total checks: 0 All validations passed! as a vacuous pass. It is not. `Total checks` is printed as `len(all_results)` in the upstream CLI, and `all_results` holds validation *issues* — so 0 means no problems were found. Planting a snippet that appears in no publication into taxonomy[0].evidence[0] produces: [ERROR] Text part not found as substring: 'ZZQQ ...' Location: taxonomy[0].evidence[0].snippet Total checks: 1 / Issues found: 1 The justfile already said this, from #257, in four lines directly above the recipe. Prose was not enough, so the claim is now a test that fails if the validator ever does accept fabricated text, plus a second test asserting a real curated snippet is still accepted - a checker that rejects everything would be just as useless and passes the first test alone. Diagnosing it turned up something the issue did not: **SupportingReference is never checked at all.** It is the range of `Discussion.evidence`, and unlike EvidenceItem its `snippet` and `reference` carry no `implements:`, so the plugin's field detection cannot see them. 11 snippets across 8 records are unvalidated, and the module's own description claims the opposite — that it "carries a verbatim snippet so the same anti-hallucination snippet-vs-cached-abstract check the Mechs already run can validate it". Not fixed here on purpose: that lives in mech_shared.yaml, which is vendored byte-identical and sha-pinned across the Mech repos by the vendored-sync job, so editing this copy alone would fail CI and violate the module's own instruction. Filed instead; the gap is pinned by a test that fails when it is closed, so nobody closes it silently. Also confirms what remains true about the truncation class: a snippet cut mid-word still passes, because a cut quote is still a substring and substring matching is what the tool promises. That is #295/#465, covered by test_snippets_are_not_truncated.py, and it is why those eight survived - not vacuity. No behavioural change; tests and comments only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #295.
Three occurrences, not two
The record cites the same
PMID:28287150snippet three times. #262 re-scoped two toPARTIAL— the snippet is the paper's generic background about Geobacter species in prior literature, not a finding about this coculture. The third stayedSUPPORT, so the record simultaneously held that the snippet establishes nothing here and that it fully supports an interaction.Worse than the issue recorded
That third occurrence also had no
explanationand a snippet truncated mid-word:#262's note on the first occurrence reads "Snippet also completed here, having previously been truncated mid-word" — the same truncation, fixed in one place and left in another.
Re-scoped to
PARTIALwith the same reasoning, snippet completed, explanation added. The interaction is unaffected either way: its other two items — the metabolic-shift quantification and the electron-uptake measurement — are what carry it, exactly as #295 predicted.A defect I introduced and the gate caught
Writing that explanation reintroduced #400's defect, in the very file I was fixing. An unquoted
#262in a plain multi-line scalar starts a YAML comment, so everything after it was swallowed and the file stopped parsing:just validatecaught it before commit. Quoted, and swept the KB for the same shape: three continuation lines contain" #", all inside quoted scalars, andvalidate-scalarsreports 0 truncated across 318 files. So the KB is clean and this was mine alone — but it is a live demonstration of why #400 is worth doing.just qcgreen.🤖 Generated with Claude Code