[infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154)

[kiki830621]

Problem

Source: surfaced during /idd-diagnose #154 sister concern surfacing (Step 3.6)

IDD plugin 目前無 systematic test framework。 每個 SKILL.md 有 dogfood scenario(README-style)但沒 executable test runner。 #154 diagnosis Block A3 需 build 9 adversarial test fixtures(R1/R2/R3 regression set)來確保 parser 不再 introduce bugs:

1. --scope=value (eq form)
2. --scope value (space form)
3. --scope as last arg (missing value)
4. --scope --body "x" (next-flag eats value)
5. --body-file=/missing/path (unreadable file)
6. Multi-line body (newline content)
7. Single-line body
8. Section with ### subsections (level-aware end)
9. Section with no closing heading (EOF as end)

問題:這 9 fixtures 無 framework → 寫 ad-hoc shell script 跑一次 → 之後 regression 無法自動 catch。 R1/R2/R3 verify 之所以失敗 = AI 寫了「應該 work」的 happy-path smoke test,沒 adversarial coverage。

Type

enhancement / infrastructure

Background

IDD plugin 既有 testing surface:

• idd-verify 是 runtime verification (issue → diagnose → impl → verify 6-AI ensemble) — catches implementation regression
• 每個 SKILL.md 有 README 風格 dogfood example — manual one-off
• 沒有 tests/idd-edit/test_parser.sh 之類 unit-level CI-able test

Closest precedent:scripts/process-attachments.sh 是 IDD helper script,但無對應 test。

Open questions

• Test framework choice:bats(bash test framework)/ plain set -e shell script / Python pytest wrapping bash?
• Trigger:single SKILL.md needs it → 順便建?或等 ≥3 SKILL.md surface 同需求?
• Scope:
  • (a) idd-edit-only fixtures(本 issue 範疇)
  • (b) IDD plugin-level test infrastructure(generalizable to all SKILL.md)
• CI integration:GitHub Actions runs on PR?或 pre-commit hook?(plugin distribution model 對 CI 期望低)

Trigger conditions (parking-lot — but lower threshold than #154 layer issue)

• Immediate:[feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 implementation Block A3 開工 → 必然需 framework choice。 可在 [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 Plan tier EnterPlanMode 階段決定 fixture 是 ad-hoc vs framework
• Threshold:≥2 SKILL.md 都需 adversarial fixtures → 升 instance idd-all: direct-commit mode 加 passive warning (dirty tree / on default branch) #2

Type

enhancement / infrastructure

Priority

P2 — higher than typical parking-lot (#154 implementation immediately need),但本身仍 follow-up scope

Refs #154 #150

---

Current Status

Phase: closed
Last updated: 2026-06-01 by idd-close

Key Decisions

• Scope B (class closure): assert-helpers.sh (assert_grep bakes in --) + lint-grep-separator.sh. Migrated idd-worktree + check-closed; pr-body got -- hardening (scanner, not assertion runner). 9b0e160, v2.79.0.
• grep--- class structurally closed ([feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 + [bug] spectra-archive-post-ic/test.sh likely has same grep -qF -- bug (sister from #154) #160 + [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156). 9 idd-edit fixtures = [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 Block A3 (deferred, can now source the lib).

Scope Changes

• (none)

Blocking

• (none)

Commits

• 9b0e160 feat: shared assert-lib + grep-separator lint to close the grep var-needle bug class ([infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156)

Verification

• PASS (focused) — 3 suites 零 regression + lint falsifiable (caught a ROOT precedence bug). Distribution sync (push + reload 2.79.0) executed.

[kiki830621]

Diagnosis

Type

enhancement / test infrastructure

Analysis — 三層 scope 不一致 + audit proportionality

三層 framing 各不同：

1. [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 as filed：IDD plugin test framework + 9 個 idd-edit parser adversarial fixtures（R1/R2/R3 regression）+ framework choice（bats / shell / pytest）+ CI 問題。是個 epic。
2. [bug] spectra-archive-post-ic/test.sh likely has same grep -qF -- bug (sister from #154) #160 衍生：「若 [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 trigger generalize → 抽 shared assertion lib .claude/scripts/lib/assert-helpers.sh」。
3. 本次 user framing：「把 grep--- class 結構性收掉：shared assert-lib + lint」。

Audit（grep--- class 真實大小）：

• spectra-archive-post-ic.sh 2 處 → 已於 [bug] spectra-archive-post-ic/test.sh likely has same grep -qF -- bug (sister from #154) #160 修。
• idd-close/SKILL.md:723、distribution-detection.md:115、pr-body-autoclose-guard/test.sh:64：grep -qE/-hE "$pattern" —— regex pattern（trap regex 不以 -- 開頭），非可觸發。
• idd-worktree/test.sh:77：grep -cxF 'literal' "$file" —— needle 是字面、$file 是 file arg，非 var-needle（false hit）。
• 結論：實質 live instance 早被 [bug] spectra-archive-post-ic/test.sh likely has same grep -qF -- bug (sister from #154) #160 收掉，其餘全是 controlled regex pattern、非可觸發。class 很小。

Test runner 現況：4 個 runner（spectra-archive-post-ic、check-closed-without-summary、pr-body-autoclose-guard、idd-worktree），各 ad-hoc，彼此共用比對邏輯不多（只 1 個用 [ = ]）。無 .claude/scripts/lib/。

Proportionality concern（核心）

user framing「shared assert-lib + lint」隱含「class 大到值得抽 lib + 遷移 runner」。但 audit 證 class 很小、runner 共用邏輯薄。為 ~2-instance class 建整套 lib + 遷移 4 runner，本身可能是過度一般化（systemic ≠ blanket）。

Scope options（待 user 拍板，見 AskUserQuestion）

• A — lint-only：建一個 check script 掃 repo 的 grep <opts> "$var" 缺 --（+ 可選的 [ = ] 對 var 的 edge）。catches 未來違規。不建 lib、不遷移。最 proportionate。
• B — lint + shared assert-lib + migrate（user 原 framing）：建 assert-helpers.sh（---safe assert_eq / assert_grep / assert_exit）+ 遷移 4 runner + lint。中等。
• C — 全 [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 framework：bats/shell framework + 9 idd-edit adversarial fixtures + lib + CI。重，是原 epic。

Complexity

Plan（design-heavy：lint 機制、是否建 lib、遷移範圍、framework choice 都是 decision；新 shared abstraction 給未來 test runner 用）—— 但 scope 需先 clarify（A/B/C 差異巨大），故先 AskUserQuestion 再進 plan。

Risks

• 過度建設（B/C 對小 class）→ 維護負擔 > 防護價值。
• 建 lib 但無人用 / runner 不遷移 → dead abstraction。

Residue

• 9 個 idd-edit adversarial fixtures（[infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 原 core）依賴 idd-edit/test.sh —— 但該檔不在 main（[feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 未 implement）。所以 fixtures 部分天然 blocked on [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154，無法在本 issue 完成。誠實標記。

Next: 先 clarify scope（A/B/C）→ /idd-plan #156

[kiki830621]

Implementation Plan (Plan tier) — scope B

Scope（user-picked B）：lint + shared assert-lib + 遷移 runner。Path: direct-commit（plugin 改動）。

關鍵發現（ground design）

• idd-worktree/test.sh:18-27 已 inline pass/fail/assert_eq/assert_exit → 即 shared-lib 的現成 template。
• 4 runner 跨兩棵樹：.claude/scripts/tests/spectra-archive-post-ic/（project-level ×1）+ plugins/issue-driven-dev/scripts/tests/{idd-worktree,check-closed-without-summary,pr-body-autoclose-guard}/（plugin-level ×3）。

Decision Points

• D1 lib 位置 = plugins/issue-driven-dev/scripts/lib/assert-helpers.sh（plugin-level）。理由：3/4 runner 在 plugin 樹，plugin 須 self-contained（會 distribute）。
• D2 project-level spectra-archive-post-ic/test.sh 不遷移：它已 [ = ]-safe（[bug] spectra-archive-post-ic/test.sh likely has same grep -qF -- bug (sister from #154) #160），且遷移會造成 .claude/ → plugins/ 跨樹 source（破 project/plugin 分離）。所以「遷移」= 3 個 plugin runner，project 那個留原樣（已安全）。這是對 user「遷移 4 個」的修正——把它縮到 3，理由在跨樹分離。
• D3 lint 範圍：掃 tracked .sh 的 grep <opts> "$var"（var-needle）缺 --；regex-pattern grep（-E "$pat"，非 -- prefixed）white-list 或只 warn。

Files & Changes

• NEW plugins/issue-driven-dev/scripts/lib/assert-helpers.sh — 抽 idd-worktree 的 pass/fail/assert_eq/assert_exit + 加 ---safe assert_grep（grep -- "$needle"）/ assert_contains / assert_file_exists / assert_file_absent / print_summary。單一 source。
• NEW plugins/issue-driven-dev/scripts/lint-grep-separator.sh — 掃 var-needle grep 缺 --，violation → exit 1。class 的 prevention。
• migrate idd-worktree/test.sh — 刪 inline helpers，改 source "$(dirname "$0")/../../lib/assert-helpers.sh"（最乾淨，本就是 template）。
• migrate check-closed-without-summary/test.sh + pr-body-autoclose-guard/test.sh — 採用 shared helpers（ad-hoc [ ] || { echo ✗ } → assert_*）。
• 版本 bump 2.78.1 → 2.79.0（minor：新 lib + lint infra）+ CHANGELOG。

Sequencing

1. 建 lib（runner 要 source 它）。2. 遷移 3 runner。3. 建 lint。4. 重跑全部 test suite + lint（regression）。

Risks & Mitigations

• 遷移破既有 test → 每遷一個立刻重跑該 suite，全綠才下一個。
• lib relative-path source 在不同 cwd 壞 → 用 $(dirname "$0") 錨定，不靠 cwd。
• dead-abstraction 風險（audit 提過）→ 由「3 runner 真的 source 它」消除；lib 只放 runner 實際用的 helper，不臆測。

Test Plan

無 unit test（test infra 本身）。Falsifiable：(a) 3 個遷移後 suite 全綠（migration 無 regression）；(b) lint 對 planted grep "$x" 缺 -- 的 fixture 回 exit 1、對乾淨 repo 回 0。

Out-of-scope

• 9 個 idd-edit adversarial fixtures（原 [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 core）—— blocked on [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154（idd-edit/test.sh 不在 main）。
• project-level spectra-archive-post-ic/test.sh 遷移（D2，已安全 + 跨樹分離）。
• 全 framework（bats/pytest/CI）—— scope C，未選。
• idd-close/distribution-detection 的 regex-pattern grep（非 var-needle，非可觸發）。

Tangential Observations (filed mid-plan)

none surfaced — audit 已在 diagnose 完成。

Next: EnterPlanMode approval → /idd-implement #156

[kiki830621]

Implementation Complete

Scope B（user-picked，含 plan D2 對「遷移 4」的修正 + implement 期再修正到「遷移 2 + 硬化 1」）。

Checklist

• NEW lib/assert-helpers.sh — 兩 family（value-comparison assert_eq/assert_exit + command-success require/refute/assert_true）+ filesystem + print_summary + class-closing assert_grep/refute_grep（內部 bake grep -F -- "$needle" → -- prefixed needle 不可能被誤判）→ 9b0e160
• NEW lint-grep-separator.sh — 掃 tracked *.sh 的 needle-position bare-var grep 缺 --，violation exit 1，# lint-ok: grep-sep 逃生口 → 9b0e160
• migrate idd-worktree/test.sh — source lib（刪 inline helpers，本就是 template）→ 34/0
• migrate check-closed-without-summary/test.sh — require/refute/assert_grep（--state closed 等 needle 現由 assert_grep bake --）→ 11/0
• pr-body-autoclose-guard/test.sh — 不 force-migrate（它是 scanner 非 assertion runner），改 in-place -- 硬化 2 個 regex grep → PASS
• 版本 2.78.1 → 2.79.0 + CHANGELOG → 9b0e160

idd-close 會 refuse 關任何 checklist 還有 - [ ] 的 issue。

Changes

• 9b0e160: feat — shared assert-lib + grep-separator lint to close the grep var-needle bug class ([infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156)

Verification

• ✅ 3 suites 全綠：idd-worktree 34/0、check-closed 11/0、pr-body PASS（+ project-level spectra-archive-post-ic 14/0 未動 sanity）。
• ✅ lint falsifiable：clean repo → PASS exit 0；planted grep "$needle" 缺 -- → FAIL exit 1（列出檔+行）；# lint-ok noqa → 豁免 exit 0。
• 🔧 verify-found-and-fixed（同 commit）：lint falsifiable test 抓到一個 ROOT operator-precedence bug（A || cd X && pwd → pwd 無條件跑 → ROOT 多一行 → git -C "$ROOT" 爛掉 → lint 靜默掃 0 檔、永遠綠）。這正是 [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 要防的 happy-path-smoke-test 失敗模式本身 —— 差點在修它的 PR 裡重演。

Scope Compliance（誠實 — 對 approved plan 的修正）

• 遷移 3 → 遷移 2 + 硬化 1：pr-body-autoclose-guard 是 file scanner（hits/missing 計數 + per-line 診斷），硬塞 assert-lib 會弄丟 per-line 診斷 = audit 警告的 over-engineering。改給它 in-place -- 硬化（同 [bug] spectra-archive-post-ic/test.sh likely has same grep -qF -- bug (sister from #154) #160）。其 regex grep 非 -- prefixed → 非可觸發。
• project-level spectra-archive-post-ic/test.sh 留原樣（plan D2：已 [ = ]-safe + 避跨樹 source）。
• 9 個 idd-edit adversarial fixtures（原 [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 epic core）+ framework choice + CI → out-of-scope（fixtures blocked on [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154）。

Distribution note

plugin 新增 lib + lint + 版本 bump → /idd-close Step 6.5 push + marketplace reload（2.79.0）。

Next: verification pending

[kiki830621]

Verify (PASS) — focused

Mode: focused（新 test infra + 遷移，blast radius = 所有 test runner → 比 chore 嚴；但核心已 falsifiable-verified，不開 full ensemble）。Diff: 9b0e160。

Acceptance

• ✅ migration 零 regression：3 suites 全綠（idd-worktree 34/0、check-closed 11/0、pr-body PASS），項目數與遷移前一致。
• ✅ lint 真的會抓（非永遠綠）：planted violation → exit 1 + 列檔行；clean → exit 0；noqa → 豁免。這個 negative test 抓到並修掉 ROOT precedence bug（否則 lint 是 always-green 假 tripwire）。
• ✅ class 結構閉合：assert_grep bake grep -F -- "$needle" → test 作者無法再忘 --；lint 擋繞過 helper 的裸 grep。雙層 = 結構性 prevention（非逐一補）。
• ✅ scope 誠實：pr-body 不 force-migrate（scanner）改 -- 硬化；spectra-archive-post-ic 留原樣（跨樹）；idd-edit fixtures deferred（blocked on [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154）。明確標於 IC。

Adversarial check — lib 設計

• assert_grep 用 stdin（printf | grep）餵 haystack → needle/haystack 都不以 argv 進 shell，-- "$needle" 讓 -- prefixed needle 當 data。✓ 對 --state closed 實測（check-closed 11/0）。
• relative-path source 用 $(dirname "${BASH_SOURCE[0]}") 錨定、不靠 cwd。✓
• dead-abstraction 風險（audit）→ lib 只放 2 runner 實際 source 的 helper，無臆測 API。

Findings

1 found（ROOT precedence）→ fixed same-commit 9b0e160。無其他 blocking / non-blocking。

Result: PASS — class 雙層結構閉合（lib assert_grep + lint）、3 suites 零 regression、lint falsifiable 確認非假綠、scope 誠實修正。

Next: /idd-close #156

[kiki830621]

Closing Summary

Problem

IDD plugin 無 systematic test framework：每個 SKILL.md 有 dogfood example 但無 executable test runner，4 個既存 runner 各 ad-hoc。並且 grep--- bug class（needle 以 -- 開頭被 grep 誤判為 flag）逐一 point-fix（#154 / #160）無結構性 prevention。

Root Cause

無共用 assertion 層 → 每個 runner 各自 inline helper、各自可能漏 --。「R1/R2/R3 verify 失敗 = AI 寫了 happy-path smoke test、無 adversarial coverage」。

Solution（scope B — class 結構閉合）

1. lib/assert-helpers.sh：抽出 runner 各自 inline 的 helper（idd-worktree 是 de-facto template）—— 兩 family + filesystem + print_summary + class-closing assert_grep/refute_grep（內部 bake grep -F -- "$needle" → -- prefixed needle 結構上不可能被誤判）。
2. lint-grep-separator.sh：tripwire 掃 tracked *.sh 的 needle-position bare-var grep 缺 --，exit 1 + noqa 逃生口。
3. 遷移：idd-worktree（source lib）+ check-closed（require/refute/assert_grep）。pr-body-autoclose-guard 是 scanner（非 assertion runner）→ in-place -- 硬化而非 force-migrate（避免弄丟 per-line 診斷 = audit 警告的 over-engineering）。

雙層 = 結構性 prevention：assert_grep 讓「忘記 --」在 test 中不可表達，lint 擋繞過 helper 的裸 grep。

Verification

3 suites 零 regression（34/0 + 11/0 + PASS）。lint falsifiable：planted violation → exit 1、clean → 0、noqa → 豁免。該 negative test 抓到並同 commit 修掉一個 ROOT operator-precedence bug（A || cd X && pwd → pwd 無條件跑 → ROOT 多一行 → lint 永遠綠）—— 正是 #156 要防的 happy-path-smoke 失敗模式本身。

Changes

• 9b0e160: feat — shared assert-lib + grep-separator lint to close the grep var-needle bug class ([infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156); bump 2.79.0

Follow-ups / Residue（誠實 — scope 修正）

• 9 個 idd-edit adversarial fixtures（原 [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156 epic core）= [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 Block A3 的 territory，blocked on [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154（idd-edit/test.sh 不在 main）。現在 assert-helpers.sh 已就位 → [feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 實作時直接 source 它寫 fixtures，不必再 inline。不重複開 issue（[feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154 已 track）。
• framework choice（bats/pytest）+ CI = scope C，未選。
• project-level spectra-archive-post-ic/test.sh 留原樣（plan D2：已安全 + 避跨樹 source）。
• grep--- class 全圖收斂：[feature] /idd-edit runtime enforcement — proper proposal after 3-iteration bash failure (R1/R2/R3 on #153) #154（idd-edit，唯一 live，已修）+ [bug] spectra-archive-post-ic/test.sh likely has same grep -qF -- bug (sister from #154) #160（spectra-archive-post-ic，defensive）+ [infra] IDD plugin test framework — adversarial fixtures for SKILL.md regression coverage (sister concern from #154) #156（lib assert_grep + lint = 結構性 prevention）。class 至此結構性收掉。