docdrift

Continuously checks whether a codebase's documentation still accurately describes its code, and flags specific drift with evidence. Runs as a GitHub Action on every PR, or as a standalone CLI.

Architecture

GitHub PR webhook
   ├─ Code parser   (tree-sitter / Python ast)        → CodeEntity[]
   └─ Doc parser    (Ollama local LLM, structured JSON) → DocClaim[]
                  ↓
         Entity matcher (rapidfuzz → pgvector fallback) → MatchedPair[]
                  ↓
         LangGraph agent: Router → Code reader → Comparator → Verdict
                  ↓
     PR comment (GitHub API) · Langfuse traces · Eval gate (GH Actions CI)

Runs on: FastAPI + Docker/K8s + pgvector

See eval/ for the golden-set precision/recall gate — this project explicitly optimizes for precision over recall, since a tool that cries wolf on every intentionally-abstract doc gets uninstalled.

Tech Stack & Rationale

Component	Choice	Why
Code parsing (Python)	ast (stdlib)	Zero deps, perfect fidelity for Python, no compilation step
Code parsing (TypeScript/JS)	tree-sitter + tree-sitter-typescript	Fast, incremental, handles real-world TS/JS with type annotations; battle-tested at GitHub/Sourcegraph
Doc extraction	Ollama (llama3.1 / qwen2.5-coder)	Local LLM, zero cost, privacy; JSON mode for structured output
Fuzzy name matching	rapidfuzz (Rust-backed)	10-100x faster than Python difflib; token_set_ratio handles "createUser" vs "create_user"
Embedding model	sentence-transformers nomic-embed-text-v1.5 (768-dim)	Strong code retrieval, runs locally on CPU/GPU; no API key needed
Vector DB	pgvector (PostgreSQL extension)	No separate infra; ACID + IVFFlat index < 100k vectors; runs in same Docker Compose
Agent orchestration	LangGraph	Explicit state machine (router → reader → comparator → verdict); checkpoints, retries, human-in-the-loop ready
LLM judgment	Ollama (same model as extraction)	Consistent reasoning; prompt enforces JSON output with confidence + suggested fix
Observability	Langfuse (optional)	Traces every comparator call (prompt, response, tokens, latency); filters by verdict for false-positive debugging
API framework	FastAPI + uvicorn	Async, OpenAPI auto-docs, dependency injection for DB sessions; slowapi for rate limiting
Rate limiting	slowapi	Token-bucket per IP; protects local model from overload
Structured logging	structlog	JSON logs with request_id correlation; integrates with Datadog/ELK/Grafana Loki
Auth	API key header (X-API-Key)	Simple, stateless, works in CI/CD and k8s; no OAuth complexity
CI/CD	GitHub Actions + Docker	Native PR diff access; docker compose for local parity; k8s manifests for prod
Eval gate	Custom precision gate	Fails CI if golden-set precision < 0.80; prevents regressions before deploy

Local Dev (Free / Open-Source)

# 1. Start Ollama (one-time)
curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull llama3.1        # or qwen2.5-coder for better code understanding

# 2. Configure
cp .env.example .env
# Edit .env if needed (defaults work for local)

# 3. Start Postgres with pgvector
docker compose up -d postgres

# 4. Install deps
pip install -e ".[dev]"

# 5. Run API
uvicorn docdrift.api.main:app --reload

Run the worker to ingest embeddings for local files:

python -m docdrift.api.worker path/to/file.py path/to/other.ts

Run the golden-set eval:

python eval/run_eval.py --fail-under 0.8

Model Options

Model	Size	Best For
qwen2.5-coder:7b	4.7 GB	Code understanding (recommended)
llama3.1:8b	4.9 GB	General + code
qwen2.5-coder:1.5b	1.1 GB	Fast, low VRAM
nomic-embed-text-v1.5	~1 GB	Embeddings (768-dim, CPU/GPU)

Project Structure

docdrift/
├── .github/workflows/
│   ├── docdrift.yml          # runs on every PR, diff-scoped for cost
│   └── eval.yml               # golden-set precision gate on push to main
├── src/docdrift/
│   ├── config.py              # env-driven settings (pydantic-settings)
│   ├── models.py              # CodeEntity, DocClaim, MatchedPair, ComparisonResult, Verdict
│   ├── extraction/
│   │   ├── code_parser.py     # ast / tree-sitter → CodeEntity[]
│   │   └── doc_parser.py      # Ollama structured-JSON extraction → DocClaim[]
│   ├── matching/
│   │   └── entity_matcher.py  # rapidfuzz + pgvector cosine similarity
│   ├── agent/
│   │   ├── state.py
│   │   ├── graph.py           # LangGraph wiring
│   │   └── nodes/
│   │       ├── router.py
│   │       ├── code_reader.py
│   │       ├── comparator.py  # core LLM judgment call (Ollama)
│   │       └── verdict.py
│   ├── output/
│   │   ├── report.py          # aggregate + filter + format
│   │   └── pr_comment.py      # GitHub API posting
│   ├── observability/
│   │   └── langfuse_setup.py
│   ├── db/
│   │   ├── models.py          # SQLAlchemy + pgvector column
│   │   └── migrations/0001_init.sql
│   ├── api/
│   │   ├── main.py            # FastAPI: POST /analyze, GET /health
│   │   └── worker.py          # async embedding ingestion (sentence-transformers)
│   └── cli.py
├── eval/
│   ├── golden_dataset.json    # 14+ labeled doc/code pairs (expand to 40-50)
│   ├── labeled_pairs/
│   └── run_eval.py            # precision/recall scoring, CI fail gate
├── k8s/
│   ├── api-deployment.yaml
│   ├── worker-job.yaml
│   └── postgres-statefulset.yaml
├── tests/
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
└── .env.example

Build Order (Suggested)

1. extraction/code_parser.py — no LLM dependency, get it solid and unit-tested first against real Python files.
2. extraction/doc_parser.py — wire the Ollama call, sanity-check JSON parsing against real README sections.
3. matching/entity_matcher.py — start with just fuzzy-name pass; embedding fallback after Postgres is up.
4. agent/ — build and test the LangGraph with 2-3 hand-picked pairs before touching FastAPI or Docker.
5. output/ + api/ — wire end-to-end locally via docker-compose.
6. eval/ — hand-label 40-50 pairs against a real target repo; get the precision-gate script running before calling it done.
7. .github/workflows/ + k8s/ — last, once the pipeline works locally.

API Usage

# Analyze a PR
curl -X POST http://localhost:8000/analyze \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-api-key" \
  -d '{
    "pr_number": 42,
    "changed_code_files": ["src/api/users.py", "src/services/auth.ts"],
    "changed_doc_files": ["docs/api.md", "README.md"],
    "post_comment": true
  }'

Response:

{
  "drift_count": 2,
  "report_markdown": "## 📄 Documentation drift detected\n\n### `docs/api.md`\n- **Line 15** — create_user takes name and email as required strings\n  - Confidence: 0.92\n  - Why: Code now requires a `role: str` parameter with no default\n  - Suggested fix: Update doc to mention required `role` parameter\n"
}

Deployment

Docker Compose (local/staging)

docker compose up -d

Kubernetes (production)

kubectl apply -f k8s/

Required secrets:

• DATABASE_URL (managed Postgres with pgvector)
• GITHUB_TOKEN (for PR comments)
• API_KEY (random 32-char string)
• Optional: LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY
• Optional: OLLAMA_HOST (if running Ollama separately)

No external LLM API keys required — runs fully offline with Ollama + sentence-transformers.

Contributing

See CONTRIBUTING.md for guidelines on:

• Code style (ruff, type hints)
• Testing (pytest, golden-set eval)
• Adding new language parsers
• Extending the agent graph
• Running the eval gate locally