CI/Linting: Future improvements for AI-generated content quality

[oskarth]

Context

This issue tracks future CI/linting improvements for ensuring quality of AI-generated documentation content. The low-hanging fruit was implemented in #57.

Tier 2: Medium Effort Improvements

1. Vale Prose Linter Integration

Effort: 4-5 hours | Impact: High

• Professional prose linter with custom IPTF rules
• Catches: terminology inconsistency, passive voice, hedging language ("might", "could potentially")
• Files needed:
  • .vale.ini - configuration
  • .vale/styles/IPTF/Marketing.yml - marketing rules
  • .vale/styles/IPTF/Terminology.yml - terms from GLOSSARY.md
  • .vale/styles/IPTF/Hedging.yml - AI hedging patterns

2. Frontmatter Schema Validation (JSON Schema)

Effort: 3-4 hours | Impact: Medium

• Full schema validation per content type using ajv
• Catches: invalid enum values, AI-invented frontmatter fields
• Add schemas: scripts/schemas/{pattern,vendor,use-case,jurisdiction}.json

3. Section Structure Validation for All Content Types

Effort: 2-3 hours | Impact: Medium

• Extend current pattern-only validation to vendors, use-cases, approaches, jurisdictions
• Define required sections per type in the validation script

4. GLOSSARY.md Term Consistency Checker

Effort: 4-5 hours | Impact: High

• Parse GLOSSARY.md, check all files use canonical spellings
• Catches: "zk-SNARK" vs "zkSNARK", "DVP" vs "DvP"
• Script: scripts/check-terminology.js

Tier 3: Higher Effort Improvements

5. Pre-commit Hooks

Effort: 3 hours | Impact: High

• Use husky + lint-staged for fast local checks before commit
• Run frontmatter, section structure, marketing checks on staged files

6. LLM-Based Content Review (On-Demand)

Effort: 8-12 hours | Impact: High

• Script using Claude API for:
  • Verifying factual claims against GLOSSARY.md
  • Checking ERC/EIP numbers are real
  • Flagging potential hallucinations
  • Consistency with related documents
• Not for CI (cost/latency), but for maintainer review

Priority Order

1. Vale prose linter (high ROI for terminology + style)
2. Glossary term consistency (catches common AI issues)
3. Section validation for all types
4. Frontmatter schemas
5. Pre-commit hooks
6. LLM review tool

Related

• PR feat(ci): enhance linting with link validation, word limits, and marketing checks #57 - Low-hanging fruit implementation

[oskarth]

Done! PR #68 adds the full AI content quality guardrails:

Validation tools:

• npm run validate - Frontmatter JSON Schema validation for patterns, use cases, vendors, jurisdictions
• npm run check-terms - GLOSSARY.md terminology consistency checker
• npm run lint:vale - Vale prose linter with custom IPTF styles
• npm run llm-review <file> - On-demand Claude-powered content review

CI integration:

• GitHub Actions workflow runs validate, check-terms, and Vale on every push
• Husky pre-commit hooks run checks locally before commit

Documentation:

• README.md updated with Development section showing how to run checks

Follow-up: Created #67 to track fixing the ~300 warnings found by the new guardrails in existing content.