[P1] issue_formatter emits Acceptance Criteria that issue_format.py can never accept — generated issues are unprocessable by construction

[stranske]

Why (verified evidence)

scripts/langchain/issue_formatter.py emits ## Acceptance Criteria by copying the source section through unchanged, but .github/scripts/issue_format.py applies its verification-gate check only to that section. For any source whose acceptance criteria are prose, the formatter therefore produces a body that the guard is guaranteed to reject — and the optimizer's format phase, which re-runs the same formatter, is guaranteed to keep producing a rejected body. That is an unbounded agents-issue-format-guard -> agents-issue-optimizer -> issues.edited loop, filed by the fleet's own generator.

• scripts/langchain/issue_formatter.py:346 — acceptance_lines = _normalize_checklist_lines(sections["acceptance"]). Normalisation only reshapes bullets into - [ ] checkboxes; it never inspects or adds a verification gate.
• scripts/langchain/issue_formatter.py:357 — when the source has no acceptance content the placeholder is - [ ] _Not provided._, which also fails the gate. Both the populated and the empty path emit a non-conforming section.
• scripts/langchain/issue_formatter.py:376-378 — that text is emitted verbatim under ## Acceptance Criteria.
• templates/consumer-repo/.github/scripts/issue_format.py:248-251 — acceptance_at = _find(body, REQUIRED["Acceptance Criteria"]) then if not GATE.search(acceptance). The gate is searched only in the Acceptance Criteria section.
• scripts/langchain/issue_formatter.py:387-391 — _formatted_output_valid checks only that the strings ## Tasks and ## Acceptance Criteria are present. The formatter's own validity check cannot detect the defect, so a body that can never pass is returned as valid.

Contract being violated: templates/consumer-repo/docs/AGENT_ISSUE_FORMAT.md requires at least one acceptance criterion naming a concrete test, smoke test, or documented live-verification gate, and its pre-submit checklist says a failing body "is not Ready; fix the body (do not file it)".

Observed consequence — stranske/Fine-Art-Archive#464. Running the canonical validator against that live body:

CANONICAL validator vs #464 body -> ok: False
  problem: `Tasks` must name a concrete file, symbol, path, config key, job, or command.
  problem: `Acceptance Criteria` names no test, runnable command or observable verification gate — Definition of Ready / Quality Bar §2 requires one.
  GATE in Acceptance Criteria: False
  GATE in Tasks            : True

The GATE in Tasks: True line is the crux: the generated body does carry five (verify: ...) hints, but they are all under ## Tasks, where the validator never looks. #464 accumulated 993 comments and ~868 workflow runs over ~20 hours and was only stopped by hand-applying agents:auto-pilot-pause.

Scope

scripts/langchain/issue_formatter.py (the generator) and its contract with .github/scripts/issue_format.py (the validator). The goal is that a body the formatter returns as valid is a body the validator accepts, so the fleet's own generator cannot file work that the format guard must reject.

Tasks

• In scripts/langchain/issue_formatter.py, after building acceptance_lines at :346, detect whether the assembled Acceptance Criteria section satisfies the same gate the validator applies. Import and reuse the single fleet definition rather than re-implementing the pattern: .github/scripts/issue_format.py is documented in .github/sync-manifest.yml:226 as the "Single fleet definition of agent-processable ... Do not fork per repo."
• When the gate is absent, append one explicit live-verification criterion to the Acceptance Criteria section naming the exact command, the exact observable response, and where evidence is captured — rather than leaving prose that cannot pass. Docs-only sources are covered by AGENT_ISSUE_FORMAT.md's live-verification substitution, so they get a gate, not an exemption.
• Change _formatted_output_valid at scripts/langchain/issue_formatter.py:387-391 so section presence is not the whole check: run the body through the shared validator and return False when it does not pass, so a non-conforming body cannot be returned as valid.
• Make the failure path fail closed: when the formatter cannot produce a conforming body, surface a non-zero/False result to the caller so the opener does not file it, instead of emitting a body that guarantees a guard loop.

Acceptance Criteria

• pytest tests/langchain/test_issue_formatter.py::test_acceptance_criteria_always_carries_a_verification_gate passes. The test feeds the formatter a source whose acceptance criteria are pure prose with (verify: ...) hints present only under Tasks (the exact #464 shape), then asserts the formatted output satisfies issue_format.validate(...).ok is True and that GATE.search(<Acceptance Criteria section>) is truthy.
• Deliberate-break demonstration: restore acceptance_text to the raw join_or_placeholder(acceptance_lines, ...) pass-through at scripts/langchain/issue_formatter.py:357, confirm test_acceptance_criteria_always_carries_a_verification_gate FAILS on the GATE.search assertion, then revert and confirm it passes again.
• python3 .github/scripts/issue_format.py <formatter output> exits 0 for every fixture in the formatter test suite, including the docs-only fixture.

Non-Goals

• Do NOT relax issue_format.py to accept a gate found under ## Tasks. AGENT_ISSUE_FORMAT.md deliberately requires the gate to be an acceptance criterion; moving the goalposts would make every generated issue "pass" while leaving them unverifiable.
• Do NOT add a docs-only or "generated body" exemption to the validator. The contract already specifies the live-verification substitution for non-test-shaped work.
• Do NOT modify agents-issue-format-guard.yml or agents-issue-optimizer.yml here — the loop-safety defects in those two files are tracked separately.
• No scaffolding / TODO-only changes; every task above is a concrete edit verified by the gate.

Implementation Notes

.github/sync-manifest.yml:226 states that issue_format.py is the single fleet definition of agent-processable and must not be forked per repo, so the formatter should import or invoke that module rather than re-implementing the GATE pattern. Note that templates/consumer-repo/.github/scripts/issue_format.py:234-247 also enforces that Tasks name a concrete file, symbol, path, config key, job or command — a second check the generator currently cannot see, and worth covering by the same shared-validator call.

Surfaced while diagnosing a 993-comment runaway loop on stranske/Fine-Art-Archive#464. Verified by reading issue_formatter.py:346-391 and issue_format.py:248-251 on origin/main, and by executing the canonical validator against the live #464 body.

[stranske]

Reopened after GitHub auto-closed it with merged PR #2999. The closer is awaiting the durable verify:compare provider report before final disposition; the implementation PR is labeled to trigger that sequence.

[stranske]

Closing after merged PR #2999 verify:compare provider report (openai PASS 86%) plus closer audit of anthropic CONCERNS against main.

Evidence: PR comment on #2999 (this round). Core defect fixed: formatter reuses fleet issue_format GATE/validate, injects live-verification AC when Tasks carry verify hints, and refuses to treat non-conforming LLM output as valid. Remaining anthropic notes were unchecked PR checklists + tests/langchain vs landed tests/scripts path naming — not missing implementation.