Skip to content

MCP: loopover_check_test_evidence ignores free-text tests evidence its sibling tools already honor #6618

Description

@JSONbored

Context

packages/loopover-engine/src/signals/test-evidence.ts exports hasLocalTestEvidence(input: { tests?: string[]; testFiles?: string[] }), which returns true when EITHER the caller supplies at least one free-text tests entry (e.g. a description like "go test ./internal/entity") OR a testFiles path that isTestPath recognizes. This is the established, repo-wide way multiple signals decide "does this change carry real test evidence":

  • packages/loopover-engine/src/signals/slop.ts:293buildSlopAssessment (backing the loopover_check_slop_risk MCP tool) calls hasLocalTestEvidence({ tests: input.tests, testFiles: input.testFiles }).
  • src/signals/boundary-test-generation.ts:107buildBoundaryTestGenerationFinding (backing the loopover_suggest_boundary_tests MCP tool) calls the same function the same way.
  • packages/loopover-engine/src/signals/engine.ts:2697 — another preflight-style signal does the same.

Correspondingly, checkSlopRiskShape and suggestBoundaryTestsShape (both in src/mcp/server.ts) each declare an optional tests: z.array(z.string().max(400)).max(2000).optional() field alongside testFiles.

loopover_check_test_evidence (checkTestEvidenceShape, src/mcp/server.ts:1180-1185) is documented as being modeled on the same pattern — its own comment reads "Coverage-gap self-check (#2235): pure local-metadata, like checkSlopRisk — the agent supplies its changed paths (plus any test paths) and asks whether the change carries enough test evidence" — but its shape only has changedPaths and testFiles, with no tests field, and its handler (checkTestEvidence, src/mcp/server.ts:3640-3660) calls classifyTestCoverage(allPaths) directly rather than consulting hasLocalTestEvidence at all. As a result, a caller whose only test evidence is free-text (e.g. "ran go test ./... locally, no new file") gets an "absent" classification and a "carries no test evidence — add or update a test" guidance line from loopover_check_test_evidence, even though the exact same evidence would be correctly recognized by loopover_check_slop_risk or loopover_suggest_boundary_tests on the same input.

Requirements

  • Add an optional tests field to checkTestEvidenceShape in src/mcp/server.ts, with the same bounds as the sibling shapes: tests: z.array(z.string().max(400)).max(2000).optional().
  • Import hasLocalTestEvidence from ../signals/test-evidence in src/mcp/server.ts (it is not currently imported there).
  • In the checkTestEvidence handler (src/mcp/server.ts:3640-3660): when classifyTestCoverage(allPaths) would return "absent" but hasLocalTestEvidence({ tests: input.tests, testFiles: input.testFiles }) returns true, the tool must NOT report "absent". Override the classification to "adequate" in that case (matching hasLocalTestEvidence's binary "evidence present" semantics — "adequate" is the lowest non-"absent" band already defined by TestCoverageClassification), and set testFileCount to at least 1 in the response in that case so it is not reported as 0 alongside a non-absent classification.
  • When the override fires, the guidance array must include a distinct line clarifying the evidence came from the free-text tests field rather than a detected test-file path (do not reuse the exact "Test coverage looks strong" or ratio-based wording used for path-derived classifications).
  • The override must only apply when the underlying path-based classification is "absent""weak"/"adequate"/"strong" classifications computed from changedPaths/testFiles must be returned unchanged regardless of the tests field's contents (this tool must not become MORE lenient than the path-based signal already is once real test-file evidence exists).

Deliverables

  • tests field added to checkTestEvidenceShape in src/mcp/server.ts
  • checkTestEvidence handler updated to consult hasLocalTestEvidence and override an otherwise-"absent" classification per the Requirements above
  • New test cases added to test/unit/mcp-check-test-evidence.test.ts: (1) changedPaths with code files only, no testFiles, but a non-empty tests array → classification is "adequate", not "absent", with distinct guidance text; (2) same but tests: [] (empty array) → classification remains "absent" (matches hasLocalTestEvidence({ tests: [] }) returning false, already asserted in test/unit/test-evidence.test.ts:76); (3) a change that is already "weak" from path evidence plus a tests entry → classification stays "weak" (override does not apply above "absent")

Test Coverage Requirements

This touches src/mcp/server.ts under src/**, so the repo's 99%+ Codecov patch gate applies. Every new branch (tests-field-only evidence, empty-tests-array, override-does-not-apply-above-absent) must be covered by the three test cases listed above.

Expected Outcome

loopover_check_test_evidence recognizes the same free-text tests evidence its sibling tools (loopover_check_slop_risk, loopover_suggest_boundary_tests) already recognize via the shared hasLocalTestEvidence helper, so the same PR no longer gets a contradictory "no test evidence" verdict from one LoopOver tool while another correctly credits the same evidence.

Links & Resources

  • packages/loopover-engine/src/signals/test-evidence.ts:46-48 (hasLocalTestEvidence)
  • packages/loopover-engine/src/signals/slop.ts:293 and src/signals/boundary-test-generation.ts:107 (existing callers/precedent)
  • src/mcp/server.ts:1180-1185 (checkTestEvidenceShape, to extend) and :3640-3660 (checkTestEvidence handler, to update)
  • test/unit/mcp-check-test-evidence.test.ts (existing test file to extend)
  • test/unit/test-evidence.test.ts:75-77 (existing hasLocalTestEvidence unit tests establishing the tests: []false behavior the override must respect)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions