diff --git a/README.md b/README.md index ceffb17..4c49325 100644 --- a/README.md +++ b/README.md @@ -1,347 +1,304 @@ +

+ CodeLens +

+ +

+ PyPI + License: MIT + Python + PRs Welcome +

+ # CodeLens — AI-Native Code Intelligence -> **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** - -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 12 CLI commands, an MCP server with 12 tools, AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). - -## Features - -- **12 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (12 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 12 tools total (auto-discovered from COMMAND_REGISTRY), every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) -- **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) -- **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement -- **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) -- **Cross-File Call Graph** — Workspace-wide call graph with import resolution and bidirectional taint propagation -- **Graph Data Model (v8.2)** — True node + edge graph (`graph_nodes` + `graph_edges` SQLite tables) for structural queries: callers, callees, blast radius, circular chains. Populated during scan; `trace` engine migrated to use it by default with `--use-graph` / `--no-graph` flag for A/B testing -- **Plugin System** — 4 plugin types (rule_pack/engine/formatter/command), 3-tier discovery (local → user → built-in), OWASP Top 10 (36 rules) + Compliance (53 rules: PCI-DSS v4.0 + HIPAA) -- **VS Code Extension** — Diagnostics Provider, Code Actions, Guard hooks, Health status bar -- **CI/CD Integration** — GitHub Actions workflows, SARIF v2.1.0 output, PR decoration, `check` quality-gate command -- **Guard Command** — Pre/post-write verification designed for AI agent workflows -- **Tree-sitter Powered** — Accurate AST-based parsing for HTML, CSS, JS, TS/TSX, Rust, Python, Vue, Svelte, Blade -- **Regex Fallback Parsers** — 28 additional languages supported via regex-based parsers (C, C++, Go, Java, Kotlin, Swift, Ruby, PHP, Scala, Dart, Elixir, Lua, R, Haskell, Nim, Objective-C, GDScript, Shell, Vim, Zig, and more) -- **Framework Auto-Detection** — React/Next.js, Vue, Svelte, Tailwind CSS, Express, Fastify, Koa, Hono, Django, Flask, FastAPI, Tauri, and more -- **Incremental Scanning** — Only re-parse changed files for speed, with SQLite persistent registry storage -- **Git-Aware Re-Index (v8.2)** — `scan --incremental` uses `git diff --name-only` to enumerate exactly the files git knows changed (mtime fallback when git unavailable). `git-status` reports the HEAD/last-indexed SHA + branch + changed-files count + re-scan recommendation in one call. `diff --git-aware` shows changed files + symbols + downstream caller impact. `watch --git-mode` polls `git diff --name-only` instead of watchdog file events. All features gracefully degrade when git is unavailable -- **Workspace Auto-Detect** — No need to specify workspace path if you're already in the project -- **AI-Optimized Output** — `--format ai` (normalized schema) and `--format compact` (token-efficient single-char keys) flags for AI agent consumption -- **Auto-Fix Engine** — Confidence-scored auto-fixes with dry-run-by-default safety -- **HTML Dashboard** — Generate visual dashboards with historical trend tracking -- **Hybrid LSP Engine** — Optional LSP-enhanced deep analysis (`--deep` flag) when language servers are available -- **Security Auditing** — Detect hardcoded secrets, data flow taint analysis, CVE scanning, ReDoS regex auditing -- **Quality Scoring** — Code smells, complexity metrics, dead code detection -- **CSS Deep Analysis** — Unused variables, orphan keyframes, specificity wars, z-index abuse -- **Performance Hints** — N+1 queries, sync blocking, memory leaks, expensive renders +> **Before an AI writes a new class, id, or function, CodeLens must be checked. This is not optional.** -## Quick Start +CodeLens is code intelligence built for AI agents, not humans skimming a dashboard. It gives an agent **full visibility into a codebase before it writes a line** — preventing collisions with existing symbols, silent overwrites, security vulnerabilities, and dead code it can't see coming. One CLI, 12 focused commands, an MCP server for direct agent integration, tree-sitter parsing across 7 core languages (13 with regex fallback for 28 more), and token-efficient output modes built for high-volume agent workflows. +**Replace this:** ```bash -# Install dependencies (tree-sitter grammars + watchdog) -bash setup.sh +grep -r "handleAuth" src/ # text match, no idea who calls it, is it dead, is it safe to touch +``` -# Initialize workspace (auto-detects frameworks) -python3 scripts/codelens.py init /path/to/workspace +**With this:** +```bash +codelens search "handleAuth" . --mode symbol # exact symbol + status(active/dead) + reference count +codelens context . --check trace --name handleAuth --direction up # every real caller, cross-file, cross-language +``` -# Scan workspace and build registry -python3 scripts/codelens.py scan /path/to/workspace +--- -# Check if "btn-primary" already exists before creating it -python3 scripts/codelens.py query "btn-primary" /path/to/workspace --domain frontend +## Table of Contents -# List all dead code -python3 scripts/codelens.py list /path/to/workspace --domain all --filter dead +- [Quick Start](#quick-start) +- [Why CodeLens](#why-codelens) +- [The 12 Commands](#the-12-commands) +- [Common Workflows](#common-workflows) +- [Interpreting Output](#interpreting-output) +- [Supported Languages & Frameworks](#supported-languages--frameworks) +- [AI Agent Integration](#ai-agent-integration) +- [Architecture](#architecture) +- [Installation](#installation) +- [Honest Competitive Positioning](#honest-competitive-positioning) +- [Contributing](#contributing) -# Detect frameworks -python3 scripts/codelens.py detect /path/to/workspace -``` +--- -### Workspace Auto-Detect (v5.1+) +## Quick Start -If you omit the workspace path, CodeLens auto-detects it: +```bash +pip install codelens + +# Scan a workspace and build the graph (auto-runs on first use of any command too) +codelens scan /path/to/workspace + +# Find a symbol before creating one — does "handleAuth" already exist? +codelens search "handleAuth" /path/to/workspace --mode symbol + +# 10-second orientation on an unfamiliar codebase +codelens context /path/to/workspace + +# What's actually dead vs. what looks dead? +codelens audit /path/to/workspace --check dead-code +``` + +Omit the workspace path and CodeLens auto-detects it from your current directory: ```bash -python3 scripts/codelens.py scan # Auto-detect workspace -python3 scripts/codelens.py query "btn-primary" # Auto-detect workspace -python3 scripts/codelens.py smell # Auto-detect workspace +cd /path/to/workspace +codelens search "handleAuth" --mode symbol +codelens context ``` -### Zero-Config for AI Agents (v6+) +### Zero-config for AI agents -If no `.codelens/` registry exists, any analysis command auto-runs `init` + `scan` (capped at 3000 files to prevent timeout): +If no `.codelens/` registry exists yet, any analysis command auto-runs `scan` first — no separate init step required: ```bash -export CODELENS_AI_MODE=1 # --format ai becomes default -python3 scripts/codelens.py query "myFunction" --lite -# → Auto-init + auto-scan → then query → {found, action} +export CODELENS_AI_MODE=1 # --format ai becomes the default +codelens search "handleAuth" . --mode symbol --lite +# → auto-scans (first run only) → returns {status, found, action} ``` -## Command Reference +> **Token budget matters.** Always pass `--lite` in an agent loop — it cuts every command's output down to the fields that actually drive a decision. See [Interpreting Output](#interpreting-output). -CodeLens consolidates 78 legacy commands into **12 focused umbrella commands** (issue #195). Each umbrella command accepts a `--check ` flag to select a specific sub-analysis, or runs all sub-analyses by default. The 32 deprecated aliases retained for one version after #195 have now been removed (issue #199) — see [Deprecated Aliases](#deprecated-aliases) below. +--- -### The 12 Umbrella Commands +## Why CodeLens -| Command | Absorbs | Description | -|---------|---------|-------------| -| `scan [workspace] [--check scan\|init\|rescan]` | scan, init, rescan | Scan workspace and build registry. `--check init` writes config only; `--check rescan` is incremental. | -| `search [workspace] "query" [--mode semantic\|symbol\|regex\|graph]` | symbols, semantic-query, query-graph, search | Unified search. Default mode is semantic (TF-IDF by meaning). `--mode symbol` for exact name lookup, `--mode regex` for code search, `--mode graph` for Cypher-subset queries. | -| `context [workspace] [--check orient\|outline\|trace\|context]` | context, outline, trace, orient | Codebase & symbol context. Default `--check orient` gives a 10-second orientation brief. `--name ` required for trace/context sub-analyses. | -| `deps [workspace] [--check affected\|dependents\|circular\|import-snapshot]` | affected, dependents, circular, import-snapshot | Dependency-graph intelligence. `--check affected` takes `--files`; `--check import-snapshot` takes `--input path.codelens.gz`. | -| `audit [workspace] [--check dead-code\|complexity\|smell\|staleness\|perf-hint\|side-effect]` | dead-code, complexity, smell, staleness, god-module, perf-hint, side-effect | Code-quality audits. Default runs all checks. | -| `security [workspace] [--check secrets\|vuln-scan\|taint\|binary-scan\|regex-audit]` | secrets, vuln-scan, taint, binary-scan, regex-audit | Security & vulnerability scans. Default runs all checks. | -| `summary [workspace] [--check summary\|dashboard\|arch-metrics\|architecture]` | summary, dashboard, arch-metrics, architecture | Auto-summary, dashboards, architecture metrics. Default `--check summary` runs the legacy prioritized-findings aggregator. | -| `impact [workspace] [--check impact\|diff\|dataflow]` | impact, diff, dataflow | Change-impact & dataflow analysis. Default `--check impact` takes `--name `. | -| `api-map [workspace] [--check api-map\|graph-schema]` | api-map, routes, graph-schema | API surface & graph schema introspection. Default `--check api-map`. | -| `doctor [workspace] [--check doctor\|env-check\|lsp-status]` | doctor, env-check, lsp-status | Environment audit. Default `--check doctor` runs the full dependency audit. | -| `history [workspace] [--check history\|ownership\|git-status]` | history, ownership, git-status | Historical trends, code ownership, git scan state. Default `--check history`. | -| `graph [workspace] "Cypher query"` | query-graph (raw Cypher) | Raw Cypher-subset graph query for power users. Casual callers should prefer `search --mode graph`. | +Grep and manual reads answer "does this string exist." They don't answer the questions that actually matter before an agent writes or deletes code: -### Deprecated Aliases +| Question grep can't answer | CodeLens command | +|---|---| +| Is this symbol actually dead, or just rarely called? | `audit --check dead-code` cross-checked with `context --check trace --direction up` | +| Who calls this function, transitively, across file *and* language boundaries? | `context --check trace --name X --direction up\|down` | +| Will deleting/changing this break something? | `impact --check impact --name X` | +| Is there a real command-injection/taint path from user input to a shell call? | `security --check taint` | +| What does this codebase even look like in 10 seconds? | `context` (orient is the default) | +| Structural graph question ("all functions calling any DB write") in one call, not five chained lookups | `search --mode graph` (Cypher subset) | -All deprecated aliases have been removed in this version (issue #199, post-#195 cleanup). The 32 legacy command names that were retained as hidden aliases for one version after the #195 consolidation — `affected`, `arch-metrics`, `architecture`, `binary-scan`, `circular`, `complexity`, `dashboard`, `dataflow`, `dead-code`, `dependents`, `diff`, `env-check`, `git-status`, `graph-schema`, `import-snapshot`, `init`, `lsp-status`, `orient`, `outline`, `ownership`, `perf-hint`, `query-graph`, `regex-audit`, `secrets`, `semantic-query`, `side-effect`, `smell`, `staleness`, `symbols`, `taint`, `trace`, `vuln-scan` — are no longer registered. Invoking any of them now produces an `invalid choice` error from argparse instead of a deprecation warning. Use the 12 umbrella commands above (e.g. `codelens audit --check dead-code` instead of `codelens dead-code`). +Every answer comes from a real SQLite-backed call graph (`graph_nodes` + `graph_edges`), built once per scan and reused — not re-grepped from scratch on every query. -### Dropped Commands (removed in issue #195) +--- -The following commands were removed entirely — broken, no value, or out of scope: `adr`, `a11y`, `handbook`, `ask`, `serve`, `sessions`, `watch`, `registry-validate`, `rule-test`, `rule-validate`, `artifact-scan`, `css-deep`, `debug-leak`, `detect`, `export-snapshot`, `refactor-safe`, `resolve-types`, `stack-trace`, `migrate` (as a command — utility module kept for tests), `benchmark`, `fix`, `self-analyze`, `guard`, `llm`, `memory`. +## The 12 Commands -### Hidden Commands (pending BOS decision) +CodeLens consolidates what used to be ~78 separate commands into **12 umbrella commands** (each with `--check ` for a specific sub-mode; omit `--check` to run all sub-analyses). -The following 13 commands are not in any absorb list nor explicitly dropped. They are kept callable but hidden from `--help` pending a BOS decision on where they belong: `analyze`, `check`, `config-drift`, `deps-audit`, `entrypoints`, `lsp`, `list`, `missing-refs`, `plugin`, `query`, `state-map`, `test-map`, `type-infer`. +| Command | `--check` sub-modes | What it answers | +|---|---|---| +| `scan` | scan (default) · rescan | Build/refresh the workspace graph. Everything else depends on this having run once. | +| `search` | semantic (default) · symbol · regex · graph | The grep replacement. `pattern` comes **first**, workspace second — opposite of every other command below. See [gotcha](#a-gotcha-worth-memorizing). | +| `context` | orient (default) · outline · trace · context | Orientation, file structure, call-chain tracing, rich symbol context. | +| `deps` | affected · dependents · circular (default: all three) · import-snapshot · export-snapshot | Dependency graph: what's affected by a change, who imports what, circular imports, team snapshot sharing. | +| `audit` | dead-code · complexity · smell · staleness · perf-hint · side-effect (default: all) | Code quality. `dead-code` cross-checked against `context --check trace` before you trust it. | +| `security` | secrets · vuln-scan · taint · binary-scan · regex-audit (default: all) | Hardcoded secrets, CVE/OSV dependency scanning, AST taint analysis, ReDoS. **Taint is Python/JS/TS/TSX only** — no Rust source/sink rules yet. | +| `summary` | summary (default) · dashboard · arch-metrics · architecture | Prioritized, anti-overload findings digest. Use `--lite` — it's designed to still be big without it. | +| `impact` | impact (default) · diff · dataflow | Blast-radius analysis before you touch something. | +| `api-map` | api-map (default) · graph-schema | HTTP/IPC route inventory, auth-middleware coverage, cheap graph-shape introspection. | +| `doctor` | doctor (default) · env-check · lsp-status | Environment/dependency health check. | +| `history` | history (default) · ownership · git-status | Trend tracking across scans, git blame ownership, scan staleness vs. HEAD. | +| `graph` | — | Raw Cypher-subset query for power users. Casual callers should use `search --mode graph` instead. | -## Query Decision Rules +### A gotcha worth memorizing -| Query Result | Action | -|-------------|--------| -| `found: false` | SAFE — create new | -| `found: true` + `status: active` | EXTEND — don't overwrite | -| `found: true` + `status: dead` | ASK user — reuse or delete? | -| `found: true` + `status: duplicate_ref` | LIST all referrers first | -| `found: true` + `status: collision` | STOP — active bug, fix first | +`search` takes `pattern` first, `workspace` second. Every other command above takes `workspace` first. Getting this backwards does **not** error — the workspace path silently becomes the search pattern and you get an empty `"ok"` result. -## Impact Risk Levels +```bash +codelens search "handleAuth" . --mode symbol # correct +codelens audit . --check dead-code # different order, also correct +``` -| Risk Level | Action | -|-----------|--------| -| `critical` | DO NOT change. Report to user. | -| `high` | Warning. List all affected first. | -| `medium` | Caution. Run tests. | -| `low` | Safe, proceed. | +--- -## Interpreting Signals +## Common Workflows -- `reference_count` / caller count = **how often** something is called, not how important it is. A function called once in the payment flow can be more critical than a utility called 50×. -- To judge importance, run `trace --direction up ` to find **who** calls it, then weigh by context (payment, auth, hot path). -- Use `--format compact` for AI/script consumption (token-efficient single-char keys), `--lite` for minimal output in large repos. -- First `scan` is intentionally slower — it builds the SQLite graph. Subsequent runs are incremental (pass `--incremental` to only re-scan changed files). +```bash +# Before creating a new component/function — does it already exist? +codelens search "AdGate" . --mode symbol --lite -## Supported Languages & Frameworks +# Full 10-second orientation on a repo you've never seen +codelens context . -**Tree-sitter parsed (AST-level):** HTML, CSS, SCSS, JavaScript, TypeScript, TSX/JSX, Rust, Python, Vue SFC, Svelte, Blade +# Is this symbol safe to delete? (cross-check dead-code with trace) +codelens audit . --check dead-code --lite +codelens context . --check trace --name myOldHelper --direction up -**Regex fallback parsed (28+ languages):** C, C++, C#, Go, Java, Kotlin, Swift, Ruby, PHP, Scala, Dart, Elixir, Lua, R, Haskell, Nim, Objective-C, GDScript, Shell/Bash, Vim, Zig, and more +# What breaks if I change this? +codelens impact . --check impact --name processPayment -**Frameworks:** React/Next.js, Vue/Nuxt, Svelte/SvelteKit, Tailwind CSS, Express, Fastify, Koa, Hono, Django, Flask, FastAPI, Tauri, Drupal, pytest, poetry, setuptools, tox, sphinx, nox, hatch +# Security sweep before a release +codelens security . --check secrets +codelens security . --check vuln-scan +codelens security . --check taint -**Package Managers:** npm, yarn, pnpm, bun, cargo, pip, pipenv, poetry, go modules +# CI/CD quality gate — exits non-zero on failure +codelens check . --severity high --max-findings 50 +codelens check . --format sarif > codelens.sarif -## Architecture +# Structural query in one call instead of chaining trace+impact+context +codelens search "MATCH (f:function)-[:CALLS]->(g:function) WHERE g.name CONTAINS 'exec' RETURN f.name, g.name LIMIT 20" . --mode graph -``` -codelens/ -├── SKILL.md # Full documentation for AI agents -├── SKILL-QUICK.md # Quick reference (concise) -├── README.md # This file -├── CHANGELOG.md # Version history (top-level) -├── CONTRIBUTING.md # Contribution guidelines -├── SECURITY.md # Security policy -├── CODE_OF_CONDUCT.md # Code of Conduct -├── LICENSE.txt # MIT License -├── setup.sh # Dependency installer -├── pyproject.toml # Python package metadata -├── skill.json # Skill metadata -├── mcp_config.json # MCP server config templates (Claude, Cursor, VS Code, Continue, Cline) -├── pytest.ini # Pytest configuration -├── references/ # Detailed reference docs -│ ├── parser-rules.md # Parsing rules per language -│ ├── query-examples.md # Query usage examples -│ ├── status-codes.md # Status & flag reference -│ ├── changelog.md # Older changelog (per-version highlights) -│ └── agent-integration.md # AI agent integration guide -├── scripts/ -│ ├── codelens.py # CLI entry point (12 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (12 tools) -│ ├── registry.py # Registry read/write/build -│ ├── persistent_registry.py # SQLite persistent storage (WAL mode) -│ ├── base_parser.py # Base tree-sitter parser -│ ├── base_engine.py # Base analysis engine -│ ├── grammar_loader.py # Lazy tree-sitter grammar loader -│ ├── framework_detect.py # Framework auto-detection -│ ├── incremental.py # Incremental scan support -│ ├── edge_resolver.py # Cross-file edge resolution -│ ├── graph_model.py # Graph data model (nodes + edges) — issue #8 -│ ├── git_aware.py # Git-diff aware incremental re-index — issue #14 -│ ├── search_engine.py # Regex code search -│ ├── trace_engine.py # Call chain tracing -│ ├── impact_engine.py # Change impact analysis -│ ├── outline_engine.py # File structure outline -│ ├── missing_refs.py # CSS/HTML mismatch detection -│ ├── diff_engine.py # Registry diff/snapshots -│ ├── circular_engine.py # Circular dependency detection -│ ├── context_engine.py # Rich symbol context -│ ├── dependents_engine.py # Module import tracking -│ ├── validate_engine.py # Registry validation -│ ├── dataflow_engine.py # Data flow taint analysis -│ ├── ast_taint_engine.py # AST-based taint analysis (tree-sitter) -│ ├── crossfile_taint_engine.py # Cross-file taint propagation -│ ├── callgraph_engine.py # Workspace-wide call graph -│ ├── smell_engine.py # Code smell detection -│ ├── sideeffect_engine.py # Side-effect analysis -│ ├── refactor_safe_engine.py # Refactoring safety check -│ ├── deadcode_engine.py # Enhanced dead code detection -│ ├── stacktrace_engine.py # Error propagation simulation -│ ├── testmap_engine.py # Test coverage mapping -│ ├── configdrift_engine.py # Dependency drift detection -│ ├── typeinfer_engine.py # Lightweight type inference -│ ├── ownership_engine.py # Git blame ownership -│ ├── secrets_engine.py # Hardcoded secret detection -│ ├── entrypoints_engine.py # Entry point mapping -│ ├── apimap_engine.py # API route mapping -│ ├── statemap_engine.py # State management tracking -│ ├── envcheck_engine.py # Environment variable audit -│ ├── debugleak_engine.py # Debug code leak detection -│ ├── complexity_engine.py # Complexity scoring -│ ├── regexaudit_engine.py # Regex auditing (ReDoS) -│ ├── a11y_engine.py # Accessibility auditing (WCAG 2.1) -│ ├── vulnscan_engine.py # Vulnerability scanning -│ ├── osv_client.py # OSV.dev API client (9 ecosystems) -│ ├── perfhint_engine.py # Performance hints -│ ├── cssdeep_engine.py # Deep CSS analysis -│ ├── autofix_engine.py # Auto-fix with confidence scoring -│ ├── dashboard_engine.py # HTML dashboard generation -│ ├── history_engine.py # Historical trend tracking -│ ├── semantic_engine.py # Semantic rules engine -│ ├── hybrid_engine.py # LSP-enhanced hybrid analysis -│ ├── lsp_client.py # LSP client wrapper -│ ├── convention_engine.py # Naming convention checking -│ ├── plugin_system.py # Plugin system & marketplace -│ ├── pre_commit_hook.py # Git pre-commit hook integration -│ ├── utils.py # Shared utilities (version, helpers) -│ ├── commands/ # One file per CLI command (auto-registered, 64 commands) -│ ├── formatters/ # Output formatters (markdown, sarif, compact, graphml) -│ ├── parsers/ # Tree-sitter + fallback parsers -│ │ ├── html_parser.py, css_parser.py, js_frontend_parser.py, js_backend_parser.py -│ │ ├── rust_parser.py, python_parser.py, tsx_parser.py, ts_backend_parser.py -│ │ ├── vue_parser.py, svelte_parser.py, tailwind_detector.py, blade_parser.py -│ │ └── fallback_*.py # 28 regex-based fallback parsers (C, C++, Go, Java, ...) -│ ├── rules/ # Built-in YAML rule packs -│ │ ├── javascript_security.yaml -│ │ └── python_security.yaml -│ └── plugins/ # Built-in plugin packs -│ ├── owasp_top10/rules/owasp_top10.yaml (36 rules, A01-A10) -│ └── compliance/rules/{hipaa.yaml, pci_dss.yaml} (53 rules) -├── benchmarks/ # Benchmark suite & fixtures (clean_app + vulnerable_app) -├── tests/ # Pytest test suite -└── vscode-codelens/ # VS Code extension source +# GraphML export — opens directly in Gephi/Cytoscape/yEd/Neo4j +codelens scan . --format graphml > codelens.graphml +codelens context . --check trace --name main --format graphml > trace.graphml ``` -## Requirements +--- -- Python 3.8+ -- tree-sitter + language grammars (auto-installed by `setup.sh`) -- watchdog (optional, for file watching) -- git (optional, for ownership analysis) -- Language server (optional, for `--deep` LSP-enhanced analysis) +## Interpreting Output -## Installation +### `--lite` is the real token-budget lever -```bash -# Clone the repository -git clone https://github.com/Wolfvin/CodeLens.git -cd CodeLens +Full non-lite output on a real workspace routinely runs 10-50x larger than `--lite`. Every command supports it; coverage of dedicated (hand-tuned) reducers vs. the generic fallback is documented in [docs/agent-usage-guide.md](docs/agent-usage-guide.md). -# Run setup -bash setup.sh +### Decision rules -# Verify -python3 scripts/codelens.py --help -``` +| `search --mode symbol` result | Action | +|---|---| +| not found | Safe to create | +| found, `status: active` | Extend, don't overwrite | +| found, `status: dead` | Ask before reusing — verify with `trace` first | +| found, multiple matches | List all referrers before touching anything | -## Integration with AI Agents +| `impact` risk level | Action | +|---|---| +| `critical` | Do not change without explicit user confirmation | +| `high` | List every affected file first | +| `medium` | Proceed with test coverage | +| `low` | Safe | -CodeLens is designed to be used by AI coding agents. The full integration guide is in [references/agent-integration.md](references/agent-integration.md). +### `reference_count` is popularity, not importance -**Key principle:** Before an AI writes any new class, id, or function, it MUST query CodeLens first to check for collisions, overwrites, and dead code. +A function called once in the payment flow can be more critical than a utility called 50 times. To judge real importance: `context --check trace --name X --direction up` to see **who** calls it, then weigh by context (payment, auth, entry point). `status: dead` in `audit --check dead-code` is not automatically "safe to delete" either — cross-check the same way; entry points (HTTP handlers, CLI subcommands, exported APIs) often show zero inbound graph edges but are still critical. -### MCP Server Integration +First `scan` on a workspace is intentionally slower (builds the SQLite graph from scratch). Every subsequent scan is incremental. -CodeLens ships with a native MCP server (55 tools) for direct AI agent integration: +--- + +## Supported Languages & Frameworks + +**Tree-sitter parsed (AST-level), verified against a real 425-file polyglot Tauri+React workspace:** Rust, TypeScript, TSX/JSX, JavaScript, Python, HTML, CSS/SCSS. Also: Vue SFC, Svelte, Blade. + +**Regex fallback (28+ additional languages):** C, C++, C#, Go, Java, Kotlin, Swift, Ruby, PHP, Scala, Dart, Elixir, Lua, R, Haskell, Nim, Objective-C, GDScript, Shell/Bash, Vim, Zig, and more. + +**Frameworks auto-detected:** React/Next.js, Vue/Nuxt, Svelte/SvelteKit, Tailwind CSS, Express, Fastify, Koa, Hono, Django, Flask, FastAPI, Tauri, and more. + +**Per-language verified coverage** (dead-code accuracy, taint gaps, trace behavior) is documented in detail in [docs/agent-usage-guide.md](docs/agent-usage-guide.md) — including the honest gap: `security --check taint` has zero Rust coverage today. + +--- + +## AI Agent Integration + +**Key principle:** before an AI writes any new class, id, or function, it should query CodeLens first to check for collisions, overwrites, and dead code. + +### MCP Server + +CodeLens ships a native MCP server (JSON-RPC over stdio) with **12 tools** — one per umbrella command, auto-discovered from the command registry: ```bash -# Start MCP server (JSON-RPC over stdio) -python3 scripts/codelens.py serve +codelens serve # not available — MCP tools are invoked by an MCP-aware client (Claude Desktop, Cursor, VS Code Copilot, Continue.dev, Cline), see mcp_config.json for templates ``` -Every MCP tool accepts a `format` parameter with the enum `[json, markdown, ai, sarif, compact, graphml]`. -For high-volume agent workflows, pass `format: "compact"` to cut token usage ~50%. -For graph-producing commands (`scan`, `trace`, `impact`, `circular`), pass `format: "graphml"` to emit a GraphML 1.0 XML document that opens directly in Gephi/Cytoscape/yEd/Neo4j (issue #59 Phase 3). Example: +Every MCP tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`/`graphml`). For high-volume agent workflows pass `format: "compact"` (single-char keys, ~50% smaller). For graph-shape introspection before paying tokens on a structural query, call `codelens_api_map` with `--check graph-schema` first: ```json -// tools/call with format=compact -{"name": "codelens_graph_schema", "arguments": {"workspace": "/path/to/proj", "format": "compact"}} -// → {"s":"ok","n":1234,"e":5678,"nts":{"function":1000,"class":234},"ets":{"CALLS":5678},"ix":6} +{"name": "codelens_api_map", "arguments": {"workspace": "/path/to/proj", "check": "graph-schema", "format": "compact"}} +// → {"s":"ok","n":1234,"e":5678,"nts":{"function":1000,"class":234},"ets":{"CALLS":5678}} ``` -The new `codelens_graph_schema` tool (issue #17) returns the graph shape in one cheap call — -use it first to decide whether structural queries (callers/callees/blast-radius) will return -meaningful results before paying tokens for them. +See [mcp_config.json](mcp_config.json) for Claude Desktop, Cursor, VS Code Copilot, Continue.dev, and Cline configuration templates. -See `mcp_config.json` for Claude Desktop, Cursor, VS Code Copilot, Continue.dev, and Cline configuration templates. - -### Guard Hooks for AI Agents +### CI/CD Quality Gate ```bash -# Pre-write safety check -python3 scripts/codelens.py guard /path/to/workspace --pre --file src/new_module.py +# Exits non-zero on failure — wire into CI +codelens check . --severity high --max-findings 50 -# Post-write verification -python3 scripts/codelens.py guard /path/to/workspace --post --file src/new_module.py +# SARIF for GitHub Advanced Security / VS Code +codelens check . --format sarif > codelens.sarif ``` -### CI/CD Quality Gate +### Plugin System ```bash -# Quality gate — exits non-zero on failure (use in CI/CD pipelines) -python3 scripts/codelens.py check /path/to/workspace --severity high --max-findings 50 +codelens plugin list +# Built-in: owasp_top10 (36 rules, A01-A10) + compliance (53 rules: PCI-DSS v4.0 + HIPAA) +``` + +--- -# SARIF output for GitHub Advanced Security / VS Code -python3 scripts/codelens.py check /path/to/workspace --format sarif > codelens.sarif +## Architecture -# GraphML export — opens in Gephi/Cytoscape/yEd/Neo4j (issue #59 Phase 3) -python3 scripts/codelens.py scan /path/to/workspace --format graphml > codelens.graphml -python3 scripts/codelens.py trace main /path/to/workspace --format graphml > trace.graphml -python3 scripts/codelens.py impact my_function /path/to/workspace --format graphml > impact.graphml -python3 scripts/codelens.py circular /path/to/workspace --format graphml > cycles.graphml +``` +codelens/ +├── SKILL.md / SKILL-QUICK.md # Full / quick reference for AI agents +├── README.md # This file +├── docs/ +│ ├── agent-usage-guide.md # Verified per-language coverage, --lite reducer coverage, known gaps +│ └── design/ # Design docs (one per feature-class PR, issue-numbered) +├── references/ # parser-rules, query-examples, status-codes, agent-integration +├── scripts/ +│ ├── codelens.py # CLI entry point +│ ├── mcp_server.py # MCP JSON-RPC server (12 tools) +│ ├── commands/ # One file per CLI command + per-umbrella --check sub-mode +│ ├── *_engine.py # Analysis engines (taint, callgraph, deadcode, secrets, ...) +│ ├── parsers/ # Tree-sitter + 28 regex fallback parsers +│ ├── formatters/ # json, markdown, ai, sarif, compact, graphml +│ ├── graph_model.py # graph_nodes + graph_edges SQLite schema +│ └── plugins/ # owasp_top10, compliance rule packs +├── benchmarks/ # Benchmark suite & fixtures +├── tests/ # pytest suite +└── vscode-codelens/ # VS Code extension source ``` -### Plugin System +## Installation ```bash -# List installed plugins -python3 scripts/codelens.py plugin list +pip install codelens +codelens --help +``` -# Built-in plugins (already shipped): -# - owasp_top10 (36 OWASP Top 10 rules, A01-A10) -# - compliance (53 rules: PCI-DSS v4.0 + HIPAA Security Rule) +For local development against the source checkout: -# Search registry (future marketplace) -python3 scripts/codelens.py plugin search "sql injection" +```bash +git clone https://github.com/Wolfvin/CodeLens.git +cd CodeLens +bash setup.sh +pip install -e . +codelens --help ``` +**Requirements:** Python 3.8+. tree-sitter grammars auto-installed by `setup.sh`. `watchdog` optional (file watching), `git` optional (ownership analysis), a language server optional (`--deep` LSP-enhanced analysis). + +--- + ## Honest Competitive Positioning -CodeLens excels in **AI-native code intelligence** — a niche where MCP integration, guard hooks, and AI-optimized output matter most. Here is an honest assessment vs established tools: +CodeLens excels in **AI-native code intelligence** — a niche where MCP integration and AI-optimized output matter most. Here's an honest assessment against established tools: | Dimension | CodeLens | SonarQube | CodeQL | Semgrep | -|-----------|:--------:|:---------:|:------:|:-------:| +|---|:---:|:---:|:---:|:---:| | AI Agent Integration | **8** | 4 | 3 | 5 | | Frontend Breadth | **8** | 6 | 3 | 5 | | MCP / AI-Native Design | **9** | 2 | 2 | 3 | @@ -353,24 +310,20 @@ CodeLens excels in **AI-native code intelligence** — a niche where MCP integra | Live CVE Scanning | 7 | 9 | 3 | **8** | | Cross-File Analysis | 6 | 8 | **10** | 7 | -**Our genuine strengths:** AI-native design, frontend analysis breadth, MCP integration, guard for AI workflows. - -**Where we lag:** Community ecosystem, IDE marketplace presence, deep abstract interpretation (CodeQL's domain), enterprise CI/CD integrations. +**Genuine strengths:** AI-native design, frontend analysis breadth, MCP integration. +**Where we lag:** community ecosystem, IDE marketplace presence, deep abstract interpretation (CodeQL's domain). +**Goal:** the best code intelligence tool for AI agent workflows — not a SonarQube replacement. -**Our goal:** Be the best code intelligence tool for AI agent workflows, not a SonarQube replacement. +--- ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. - -## Security - -See [SECURITY.md](SECURITY.md) for reporting vulnerabilities. +See [CONTRIBUTING.md](CONTRIBUTING.md). Security issues: see [SECURITY.md](SECURITY.md). ## License -MIT License — see [LICENSE.txt](LICENSE.txt) +MIT — see [LICENSE.txt](LICENSE.txt). ## Changelog -See [CHANGELOG.md](CHANGELOG.md) for version history (top-level) and [references/changelog.md](references/changelog.md) for older per-version highlights. +[CHANGELOG.md](CHANGELOG.md) (current) · [references/changelog.md](references/changelog.md) (older per-version highlights). diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index 8e9ed09..4f0112a 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -2,192 +2,151 @@ **MUST activate before writing/editing/deleting any class, id, or function.** -> Read THIS FILE FIRST. All commands: auto-detect workspace, auto-setup, smart `--top 20`, `--lite`, `--max-tokens N`, `--format ai`, `--format compact` (issue #17). +> Read THIS FILE FIRST. All commands: auto-detect workspace, auto-setup, `--lite`, `--max-tokens N`, `--format ai`, `--format compact`. +> +> **Architecture note:** CodeLens consolidated ~78 legacy commands into **12 umbrella commands**, each with `--check `. If you remember standalone `query`/`trace`/`dead-code`/`secrets`/`init`/`serve`/`guard` — those no longer exist as top-level commands. See [SKILL.md](SKILL.md#architecture--read-this-first-if-you-know-an-older-codelens) for the full old→new mapping. ## Zero-Config Usage ```bash -CLI="python3 /path/to/codelens/scripts/codelens.py" -export CODELENS_AI_MODE=1 # Optional: --format ai becomes default -$CLI query "myFunction" --lite # → {found, action} -$CLI smell # → Auto --top 20, sorted by severity -$CLI complexity --top 5 --lite # → Top 5 most complex, minimal output -$CLI trace "main" --direction down --format compact # → token-efficient single-char keys (issue #17) -$CLI graph-schema # → graph shape: nodes/edges/types in ~50 bytes (issue #17) -$CLI list --limit 5 --offset 10 --format compact # → paginated + compact +export CODELENS_AI_MODE=1 # Optional: --format ai becomes default +codelens search "myFunction" . --mode symbol --lite # → symbol status + ref count +codelens audit . --check smell --lite # → health_score, top findings by severity +codelens audit . --check complexity --top 5 --lite # → top 5 most complex, minimal output +codelens context . --check trace --name main --direction down --format compact +codelens api-map . --check graph-schema # → graph shape: nodes/edges/types in ~50 bytes +codelens search "pattern" . --mode regex --limit 5 --offset 10 --format compact ``` -**Auto-setup** caps at 3000 files to prevent timeout. For full analysis: `$CLI scan` manually. +**Auto-setup** caps at 3000 files to prevent timeout. For full analysis: `codelens scan` manually (no cap). -## AI Flags (work with ANY command) +## AI Flags (work with every command) | Flag | Effect | -|------|--------| -| `--top N` | Limit list to N items (sorts by relevance: severity/complexity). Smart default: 20. Override: `--top 0` unlimited | +|---|---| +| `--top N` | Limit list to N items (sorts by relevance: severity/complexity) | | `--lite` | Command-specific minimal output (see table below) | | `--max-tokens N` | Auto-truncate to fit ~N tokens | | `--format ai` | Normalized: `{stats, items[], truncated, recommendations}` | -| `--format compact` | Token-efficient: single-char keys + abbreviated types (issue #17). ~50% smaller than `json`. Best for high-volume MCP tool calls | -| `--format sarif` | SARIF v2.1.0 output for GitHub Advanced Security / VS Code | -| `--format graphml` | GraphML 1.0 XML for graph-producing commands (`scan`, `trace`, `impact`, `circular`). Opens in Gephi/Cytoscape/yEd/Neo4j. Other commands emit a single-node placeholder (issue #59 Phase 3) | -| `--limit N` / `--offset N` | Pagination on list-type commands (`list`, `search`, `trace`, `symbols`, `outline`). Default limit=20 (issue #17). `--top N` is an alias for `--limit N --offset 0` | -| `--deep` | Enable LSP-enhanced deep analysis (requires language server; check with `lsp-status`) | -| `--db-path PATH` | Custom SQLite database path (default: `.codelens/codelens.db`) | -| `--diff-base REF` | Git ref (branch/tag/SHA/`HEAD~1`) to diff against. Only findings from files changed relative to REF are reported. Empty diff → early exit. Useful for CI PR checks. Works on all analysis commands (issue #157) | +| `--format compact` | Single-char keys + abbreviated types, ~50% smaller than `json`. Best for high-volume MCP calls | +| `--format sarif` | SARIF v2.1.0 for GitHub Advanced Security / VS Code | +| `--format graphml` | GraphML 1.0 XML for graph-producing commands (`scan`, `context --check trace`, `impact`, `deps --check circular`). Opens in Gephi/Cytoscape/yEd/Neo4j | +| `--limit N` / `--offset N` | Pagination on list-type results. Default limit=20 | +| `--deep` | LSP-enhanced deep analysis (requires a language server; check with `doctor --check lsp-status`) | +| `--db-path PATH` | Custom SQLite database path (default `.codelens/codelens.db`) | +| `--diff-base REF` | Git ref to diff against — only findings from changed files reported (pre-filter, useful for CI PR checks) | ### Lite Mode Per Command | Command | `--lite` returns | -|---------|------------------| -| `query` | `{status, found, action, action_reason}` | -| `impact` / `refactor-safe` | `{status, risk, action}` | -| `smell` | `{health_score, total_findings, action, top_findings[], stats}` | -| `complexity` | `{stats, top_complex[], high_complexity_count}` | -| `dead-code` | `{removal_safety, recommended_action, stats, top_items[], total_dead}` | -| `debug-leak` | `{stats, top_leaks[], leaks_total}` | -| `perf-hint` | `{risk, stats, top_hints[], hints_total}` | -| `secrets` | `{risk, action, stats, top_findings[]}` | -| `a11y` / `css-deep` / `regex-audit` | `{risk, stats, top_items[], recommendations[]}` | -| `vuln-scan` | `{risk, stats, findings[], osv_stats, cache_info{last_refresh, age_hours, ttl_hours, is_stale, stale_packages[]}, recommendations[]}` — `cache_info.is_stale` tells agents whether to re-run with `--refresh` (issue #30) | -| `taint` | `{status, stats, top_violations[], recommendations[]}` | -| `guard` | `{status, risk, action, blocked_reason?}` | -| `check` | `{status, exit_code, total_findings, critical_count}` | -| Other | `{status, stats, top 5 items, recommendations}` | - -## Query Decision Rules - -| Result | Action | -|--------|--------| -| `found: false` | CREATE — safe to write new | -| `found: true` + `active` | EXTEND — don't overwrite | -| `found: true` + `dead` | ASK user — reuse or delete? | -| `found: true` + `duplicate_ref` | LIST_FIRST — show all referrers | -| `found: true` + `collision` | STOP — active bug, fix first | +|---|---| +| `impact --check impact` / `--check diff` | `{status, risk, action}` | +| `audit --check smell` | `{status, health_score, total_findings, action, top_findings[], stats}` | +| `audit --check complexity` | `{status, stats, top_complex[], high_complexity_count}` | +| `audit --check dead-code` | `{status, removal_safety, recommended_action, stats, top_items[], total_dead}` | +| `security --check secrets` | `{status, risk, action, stats, top_findings[]}` | +| `security --check taint` | `{status, risk, stats, top_findings[], recommendations}` | +| `security --check vuln-scan` | `{status, risk, stats, findings[], recommendations}` | +| `summary` | `{status, workspace, identity, frameworks, recommendations, findings[]}` — each finding's items capped to 3 | +| `history` | `{status, workspace, snapshots, latest{...}, trends, deltas}` | +| Other | generic fallback: `{status, stats, top 5 items, recommendations}` | + +## Search Decision Rules + +| `search --mode symbol` result | Action | +|---|---| +| not found | CREATE — safe to write new | +| found + `active` | EXTEND — don't overwrite | +| found + `dead` | ASK user — reuse or delete? (cross-check with `trace` first) | +| found + `duplicate_ref` | LIST_FIRST — show all referrers | +| found + `collision` | STOP — active bug, fix first | ## Trigger Map | Intent | Command | -|--------|---------| -| Create/edit/delete code | `query` → write → `scan --incremental` | -| "what changed?" | `diff --git-aware` | -| "do I need to re-scan?" | `git-status` | -| "does this exist?" | `query --lite` | -| "who calls this?" | `trace --direction up` | -| "safe to delete?" | `impact` → `dead-code` | -| "safe to rename?" | `refactor-safe` | -| "production ready?" | `smell` → `complexity` → `debug-leak` → `secrets` | -| "security audit" | `secrets` → `dataflow` → `env-check` → `vuln-scan` | -| "are CVE results fresh?" | `vuln-scan` → check `cache_info.is_stale` → if stale, re-run `vuln-scan --refresh` or `vuln-scan --max-age 6h` (issue #30) | -| "taint analysis" | `taint` (AST) or `dataflow` (cross-file) | -| "what to refactor?" | `smell` | -| "too complex?" | `complexity` | -| "performance?" | `perf-hint` → `circular` | -| "cleanup before deploy" | `debug-leak` → `dead-code` → `secrets` | -| "CSS issues?" | `css-deep` → `missing-refs` | -| "accessible?" | `a11y` | -| "project overview" | `architecture --lite` (single call, <1k tokens) or `summary` / `handbook` (deeper) | -| "fix automatically" | `fix --apply` (dry-run by default) | -| "show dashboard" | `dashboard` | -| "trend over time" | `history` | -| "run everything" | `analyze` | +|---|---| +| Create/edit/delete code | `search --mode symbol` → write → `scan --incremental` | +| "what changed?" | `impact --check diff` | +| "do I need to re-scan?" | `history --check git-status` | +| "does this exist?" | `search --mode symbol --lite` | +| "who calls this?" | `context --check trace --direction up` | +| "safe to delete?" | `impact --check impact` → `audit --check dead-code` | +| "production ready?" | `audit --check smell` → `audit --check complexity` → `security --check secrets` | +| "security audit" | `security --check secrets` → `impact --check dataflow` → `security --check vuln-scan` | +| "taint analysis" | `security --check taint` (AST, single-file) or `impact --check dataflow` (cross-file) | +| "what to refactor?" | `audit --check smell` | +| "too complex?" | `audit --check complexity` | +| "performance?" | `audit --check perf-hint` → `deps --check circular` | +| "cleanup before deploy" | `audit --check dead-code` → `security --check secrets` | +| "project overview" | `context` (10-second orient, default sub-mode) or `summary --lite` (findings digest) | | "CI/CD gate" | `check --severity high` | -| "manage plugins" | `plugin list` / `plugin install` | -| "MCP server" | `serve` | -| "pre/post-write hook" | `guard --pre` / `guard --post` | -| "don't know which command" | `ask "question"` | -| "LSP servers available?" | `lsp-status` (issue #33: `--lsp-status` top-level flag is an alias — both produce the identical payload, and the MCP `codelens_lsp_status` tool uses the same subcommand path) | +| "manage plugins" | `plugin list` | +| "structural query in one call" | `search --mode graph` (Cypher subset) or `graph "cypher"` for raw power-user queries | +| "LSP servers available?" | `doctor --check lsp-status` | +| "share graph with teammate" | `deps --check export-snapshot` → they run `deps --check import-snapshot` | ### Disambiguation | You want | Use | Not | -|----------|-----|-----| -| "Name exists?" | `query` | `symbols` (query checks status+action) | -| "Who calls?" | `trace` | `context` (trace goes deep) | -| "Quick symbol info" | `context` | `trace` (context is 1-level) | -| "Find text in code" | `search` | `symbols` (search is regex on files) | -| "Quality check" | `smell` | `complexity` (smell = 10 categories) | -| "Complexity score" | `complexity` | `smell` (complexity = metrics) | -| "Pre-delete check" | `impact` | `dead-code` (impact shows breakage) | -| "Find unused code" | `dead-code` | `impact` (dead-code finds unused) | -| "Security in my code" | `secrets` | `vuln-scan` (vuln-scan checks deps) | -| "Dependency CVEs" | `vuln-scan` | `secrets` (secrets finds hardcoded) | -| "Project identity" | `handbook` | `summary` (summary = findings) | -| "AST taint (precise)" | `taint` | `dataflow` (dataflow is cross-file, regex-aware) | -| "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | -| "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | - -## All 12 Commands - -### Setup & Lifecycle (8+) -`init` · `scan [--incremental] [--max-files N] [--full]` · `registry-validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` (issue #33: `codelens --lsp-status` top-level flag is an alias of `codelens lsp-status` — both delegate to `hybrid_engine.get_lsp_status()` and return the identical payload) - -### Pre-Write Safety (5) -`query "name" [--domain ...] [--fuzzy]` · `impact "name" [--action modify|delete]` · `refactor-safe "name" [--action rename|move]` · `guard (--pre|--post) --file PATH` · `check [--severity ...] [--max-findings N]` - -### Navigation (11) -`architecture [--lite] [--no-cache]` · `summary [--focus security|quality|architecture|all] [--detail minimal|standard|full]` · `context "name"` · `trace "name" [--direction up|down|both] [--limit N] [--offset N]` · `search "pattern" [--limit N] [--offset N]` · `symbols "name" [--fuzzy] [--limit N] [--offset N]` · `outline [--file path] [--limit N] [--offset N]` · `dependents "file"` · `list [--filter ...] [--limit N] [--offset N]` · `ask "question"` · `diff [--git-aware]` - -### Architecture (10) -`entrypoints` · `api-map` · `state-map` · `detect` · `handbook` · `diff [--git-aware]` · `dashboard` · `history` · `graph-schema` · `resolve-types` - -### Security (6) -`secrets [--severity ...]` · `taint` (AST-based) · `dataflow [--source ...] [--sink ...]` (cross-file) · `vuln-scan [--offline] [--osv-ttl N] [--refresh] [--max-age Nh]` (OSV.dev + native audit; `--refresh` bypasses cache, `--max-age Nh` overrides per-run TTL, `cache_info` in output signals staleness — issue #30) · `deps-audit [--severity ...] [--ecosystem PyPI|npm|crates.io] [--offline]` (pure-Python OSV.dev dependency audit, stores findings as `dependency_vuln` graph nodes — issue #158) · `env-check [--var NAME]` - -### Quality (9) -`smell [--categories ...] [--severity ...]` · `complexity [--name FN] [--threshold N] [--sort ...]` · `dead-code [--categories ...]` · `debug-leak [--category ...]` · `circular [--domain ...]` · `missing-refs` · `side-effect [--name FN]` · `perf-hint [--severity ...] [--category ...]` · `fix [--apply]` - -### Refactoring (3) -`test-map` · `stack-trace "name"` · `config-drift` - -### Frontend (2) -`css-deep` · `a11y` - -### Advanced & RE (5) -`analyze [--focus ...] [--timeout SECS]` · `type-infer` · `ownership` · `regex-audit` · `binary-scan` · `artifact-scan [--deep]` - -### Tooling (1) -`plugin ` +|---|---|---| +| "Name exists?" | `search --mode symbol` | `search --mode semantic` (fuzzy-by-meaning, not exact) | +| "Who calls X, transitively?" | `context --check trace --direction up` | `context --check context` (single-level) | +| "Quick symbol info" | `context --check context --name X` | `context --check trace` (trace goes multi-level deep) | +| "Find literal text/regex in code" | `search --mode regex` | `search --mode symbol` (symbol is exact-name lookup, not free text) | +| "Quality check" | `audit --check smell` | `audit --check complexity` (smell = multi-category, complexity = one metric) | +| "Pre-delete check" | `impact --check impact` | `audit --check dead-code` (impact shows blast radius; dead-code shows current unused status) | +| "Security in my code" | `security --check secrets` | `security --check vuln-scan` (vuln-scan checks dependency CVEs, not your code) | +| "Dependency CVEs" | `security --check vuln-scan` | `security --check secrets` (secrets finds hardcoded keys) | +| "Auto-fix issues" | not available — CodeLens finds, it doesn't auto-fix | `check` (gates CI, doesn't fix) | + +## The 12 Umbrella Commands + +| Command | `--check` sub-modes | +|---|---| +| `scan` | scan (default) · rescan | +| `search "pattern" [workspace]` | semantic (default) · symbol · regex · graph — **pattern first, workspace second**, opposite of every command below | +| `context [workspace]` | orient (default) · outline · trace (`--name X --direction up\|down\|both`) · context (`--name X`) | +| `deps [workspace]` | affected (`--files ...`) · dependents (`--files ...`) · circular · import-snapshot (`--input path.gz`) · export-snapshot (`--output path.gz`) | +| `audit [workspace]` | dead-code · complexity · smell · staleness · perf-hint · side-effect | +| `security [workspace]` | secrets · vuln-scan · taint · binary-scan · regex-audit | +| `summary [workspace]` | summary (default) · dashboard · arch-metrics · architecture | +| `impact [workspace]` | impact (`--name X`, default) · diff · dataflow | +| `api-map [workspace]` | api-map (default) · graph-schema | +| `doctor [workspace]` | doctor (default) · env-check · lsp-status | +| `history [workspace]` | history (default) · ownership · git-status | +| `graph [workspace] "cypher query"` | — (power-user raw Cypher) | + +Plus two commands hidden from `--help` but callable directly, pending final placement (issue #200): `check [workspace] --severity ... --max-findings N` (CI/CD quality gate), `plugin list` (plugin management). **Total: 12 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) ## MCP Server (12 Tools) -Start the MCP server for AI agent integration: - -```bash -python3 scripts/codelens.py serve -``` - -Exposes 12 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): -- 50 statically-defined tools (full JSON schemas in `mcp_server.py`) -- -44 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) -- Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). -- `watch` and `serve` itself are excluded (long-running) +MCP tools are invoked by an MCP-aware client (Claude Desktop, Cursor, VS Code Copilot, Continue.dev, Cline) — there is no standalone `codelens serve` command to run yourself. Point your MCP client config at `scripts/mcp_server.py`; see [mcp_config.json](mcp_config.json) for ready-made templates. -See `mcp_config.json` for Claude Desktop, Cursor, VS Code Copilot, Continue.dev, and Cline configuration templates. +- **12 tools total** — one `codelens_` per umbrella command (e.g. `codelens_search`, `codelens_audit`, `codelens_security`), auto-discovered from `COMMAND_REGISTRY` (6 statically-defined with full JSON schemas + 6 dynamically-discovered) +- Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`/`graphml`) — use `format: "compact"` for token-efficient responses +- Long-running commands (`watch`) are excluded from MCP exposure ## Error Handling -All errors return `{status:"error", error_type, error, suggestion}`. Common patterns: +All errors return `{status:"error", error, error_type}` (some also include a `suggestion` field). Common patterns: | Condition | Recovery | -|-----------|----------| -| No `.codelens/` | Auto-init+scan (zero-config) | -| Tree-sitter missing | Regex fallback (run `setup.sh` for AST) | -| `ask` timeout (45s) | Run specific command directly | -| Any `status:"error"` | Follow `suggestion` field | +|---|---| +| No `.codelens/` registry | Auto-scans (zero-config) — no separate init step needed | +| Tree-sitter missing | Regex fallback kicks in automatically (run `setup.sh` for full AST accuracy) | +| Any `status:"error"` | Follow the `error`/`suggestion` field in the response | | Workspace invalid | Auto-detect from cwd/parent dirs | -| `analyze` engine timeout | `skipped:true` per-engine — run that command individually | -| `summary` budget exceeded | `timed_out_engines[]` — use `--detail minimal` or specific commands | -| `handbook` budget exceeded | `partial:true` — run individual commands for skipped sections | -| `guard` blocks write | Follow `blocked_reason` — fix the flagged issue before retrying | +| `search "X" "Y"` returns empty `"ok"` result unexpectedly | Check argument order — `search` is pattern-first, everything else is workspace-first | +| `--lite` output looks emptier than expected | Command may be hitting the generic fallback reducer, not a dedicated one — see Lite Mode table above | -## First-Time Setup (if zero-config fails) +## First-Time Setup ```bash -bash /path/to/codelens/setup.sh # One-time: install tree-sitter -$CLI init # Creates .codelens/ config -$CLI scan # Builds registry (~5-15s for <500 files) -$CLI query "main" # Verify: returns {found, action} -# After code changes: $CLI scan --incremental (~1-5s) +pip install codelens +codelens scan /path/to/project # Builds registry (~5-15s for <500 files); no separate init step +codelens search "main" /path/to/project --mode symbol # Verify: returns a valid JSON result +# After code changes: +codelens scan /path/to/project --incremental # ~1-5s ``` diff --git a/SKILL.md b/SKILL.md index 59450cd..8bea96e 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,12 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 12 commands for AI-powered code analysis, - security auditing, quality scoring, AST-based taint analysis, live CVE scanning, - and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 12 tools for AI agent integration. + CodeLens — AI-Native Code Intelligence. 12 commands for AI-powered code + analysis, security auditing, quality scoring, AST-based taint analysis, live CVE + scanning, and pre-write safety checks (each command is an umbrella over focused + --check sub-modes). Tree-sitter parsing for 7 core languages (Rust, TypeScript, + TSX, JavaScript, Python, HTML, CSS) plus 28+ languages via regex fallback. + MCP server exposes 12 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- @@ -15,202 +17,200 @@ Before an AI writes a new class/id/function, CodeLens must be checked. This is n **Quick command reference →** [SKILL-QUICK.md](SKILL-QUICK.md) (validated output schemas, error behavior, trigger maps) **Version history →** [CHANGELOG.md](CHANGELOG.md) +**Verified per-language coverage & known gaps →** [docs/agent-usage-guide.md](docs/agent-usage-guide.md) + +--- + +## Architecture — read this first if you know an older CodeLens + +CodeLens consolidated ~78 legacy commands into **12 umbrella commands** (issue #195/#199/#200). If you have seen CodeLens before and remember `query`, `init`, `smell`, `dead-code`, `secrets`, `trace`, `impact`, `circular`, `guard`, or `serve` as top-level commands — those no longer exist as standalone commands. They are now `--check ` flags under one of the 12 umbrellas below: + +``` +codelens query "name" . → DROPPED (no direct replacement; use `search --mode symbol`) +codelens dead-code . → codelens audit . --check dead-code +codelens secrets . → codelens security . --check secrets +codelens trace "name" . → codelens context . --check trace --name "name" +codelens impact "name" . → codelens impact . --check impact --name "name" +codelens circular . → codelens deps . --check circular +codelens init . → DROPPED (scan auto-inits) +codelens serve → DROPPED (MCP tools invoked by an MCP-aware client, not this CLI) +codelens guard --pre --file X → DROPPED +``` + +## The 12 Umbrella Commands + +| Command | `--check` sub-modes | +|---|---| +| `scan` | scan (default) · rescan | +| `search` | semantic (default) · symbol · regex · graph — **`pattern` comes first, workspace second**, opposite of every other command here | +| `context` | orient (default) · outline · trace · context | +| `deps` | affected · dependents · circular (default: all three) · import-snapshot · export-snapshot | +| `audit` | dead-code · complexity · smell · staleness · perf-hint · side-effect (default: all) | +| `security` | secrets · vuln-scan · taint · binary-scan · regex-audit (default: all) | +| `summary` | summary (default) · dashboard · arch-metrics · architecture | +| `impact` | impact (default) · diff · dataflow | +| `api-map` | api-map (default) · graph-schema | +| `doctor` | doctor (default) · env-check · lsp-status | +| `history` | history (default) · ownership · git-status | +| `graph` | — (raw Cypher; casual callers use `search --mode graph` instead) | + +Two additional commands are registered but hidden from `--help` (pending a maintainer decision on their final home, issue #200): `check` (CI/CD quality gate) and `plugin` (plugin management). Both work today — call them directly, e.g. `codelens check . --severity high`. --- ## Zero-Config for AI — Just Run Any Command -CodeLens now supports **zero-config AI usage**. If no registry exists, running any analysis command automatically triggers `init` + `scan`: +If no `.codelens/` registry exists, running any analysis command automatically triggers `scan` first: ```bash -$CLI query "myFunction" --lite -# → If no .codelens/ exists: auto-init + auto-scan → then query -# → Returns: {status:"ok", found:true|false, action:"CREATE"|"EXTEND"|"ASK"|"STOP"} +codelens search "myFunction" . --mode symbol --lite +# → If no .codelens/ exists: auto-scan → then search +# → Returns: {status:"ok", ...} — see the specific command's Lite Mode row below ``` -### AI-Optimized Flags (work with ANY command) +### AI-Optimized Flags (work with every command) | Flag | Effect | When to use | -|------|--------|-------------| +|---|---|---| | `--top N` | Limit list results to top N items (sorts by relevance first) | Large repos, token budget concerns | | `--max-tokens N` | Truncate output to fit ~N tokens | Strict context window limits | -| `--lite` | Minimal output: command-specific tailored response | Quick checks, decision-making | +| `--lite` | Minimal output: command-specific tailored response | Quick checks, decision-making — **use this by default in an agent loop** | | `--format ai` | Normalized schema: `{stats, items[], truncated, recommendations}` | Consistent parsing across commands | - -### Smart Defaults (Zero-Config Token Savings) - -- **Auto `--top 20`**: List commands (smell, complexity, dead-code, secrets, etc.) auto-apply `--top 20`. No more 1000+ item responses by default. -- **Sort-aware `--top`**: Items are sorted by relevance BEFORE truncating — severity for quality commands, cyclomatic score for complexity, effect_count for side-effect. -- **Command-specific `--lite`**: 10+ commands have tailored lite output, not just query. Each lite mode returns the most actionable subset. -- **Override**: Use `--top 0` for unlimited results, or `--top N` for any custom limit. +| `--format compact` | Single-char keys, ~50% smaller than `json` | High-volume MCP tool calls | ### Lite Mode Per Command | Command | `--lite` returns | -|---------|------------------| -| `query` | `{status, found, action, action_reason}` | -| `impact` / `refactor-safe` | `{status, risk, action}` | -| `smell` | `{status, health_score, total_findings, action, top_findings[], stats}` | -| `complexity` | `{status, stats, top_complex[], high_complexity_count}` | -| `dead-code` | `{status, removal_safety, recommended_action, stats, top_items[], total_dead}` | -| `debug-leak` | `{status, stats, top_leaks[], leaks_total}` | -| `perf-hint` | `{status, risk, stats, top_hints[], hints_total}` | -| `secrets` | `{status, risk, action, stats, top_findings[]}` | -| Other | `{status, stats, top 5 items, recommendations}` | +|---|---| +| `search --mode symbol` | full symbol result (no dedicated reducer — small payload already) | +| `impact --check impact` / `impact --check diff` | `{status, risk, action}` | +| `audit --check smell` | `{status, health_score, total_findings, action, top_findings[], stats}` | +| `audit --check complexity` | `{status, stats, top_complex[], high_complexity_count}` | +| `audit --check dead-code` | `{status, removal_safety, recommended_action, stats, top_items[], total_dead}` | +| `security --check secrets` | `{status, risk, action, stats, top_findings[]}` | +| `security --check taint` | `{status, risk, stats, top_findings[], recommendations}` | +| `summary` | `{status, workspace, identity, frameworks, recommendations, findings[]}` — each finding's `top_items` capped to 3, nested `flow_chain` stripped | +| `history` | `{status, workspace, snapshots, latest{health_score,...}, trends, deltas}` | +| Other | generic fallback: `{status, stats, top 5 items, recommendations}` — adequate, not hand-tuned | ### The One Command You Need ```bash -export CODELENS_AI_MODE=1 # Optional: makes --format ai the default -$CLI query "name" --lite # Auto-setup + minimal response = {found, action} +codelens search "handleAuth" . --mode symbol --lite ``` -If `action: CREATE` → safe to write. If anything else → check first. +If not found → safe to write. If found + `status: active` → extend, don't overwrite. If found + `status: dead` → ask before reusing. --- ## Onboarding — First-Time AI Setup -This section guides a new AI agent through the complete CodeLens setup process from zero to productive. - ### Prerequisites - Python 3.8+ installed - Target codebase accessible on filesystem -- ~50MB disk space for tree-sitter grammars (optional but recommended) -### Step 1: Install Dependencies +### Step 1: Install ```bash -bash /path/to/codelens/setup.sh +pip install codelens ``` -This installs tree-sitter and language grammar packages. If this fails or tree-sitter is unavailable, CodeLens automatically falls back to regex-based parsing — it still works, just with less precision. +(For a source checkout: `bash setup.sh && pip install -e .` — installs tree-sitter grammars. If tree-sitter is unavailable, CodeLens automatically falls back to regex-based parsing.) -### Step 2: Initialize the Workspace +### Step 2: Build the Registry ```bash -CLI="python3 /path/to/codelens/scripts/codelens.py" -$CLI init /path/to/project +codelens scan /path/to/project ``` -**What it does:** Creates `.codelens/` directory in the workspace root with config file. Auto-detects project type, framework, and language. +No separate `init` step — `scan` handles workspace detection and registry creation in one call. -**Output:** `{status:"ok", workspace, codelens_dir, config{frontend_paths, backend_paths, ignore, frameworks, ...}}` +**Timing:** <500 files: ~5-15s · 1,000-5,000 files: ~30-120s · 5,000+ files: use `--max-files 3000` to prevent timeout. -**What if the workspace path is wrong?** CodeLens auto-detects by walking up from cwd to find a project root (package.json, pyproject.toml, Cargo.toml, etc.). If it auto-detects, you see a stderr warning. +**Output:** `{status:"ok", files_scanned{...}, frontend{classes,ids}, backend{nodes,edges}}` -### Step 3: Build the Registry (REQUIRED) +### Step 3: Verify ```bash -$CLI scan /path/to/project +codelens search "main" /path/to/project --mode symbol ``` -**What it does:** Parses all source files, builds a registry of symbols (functions, classes, IDs, CSS classes) and their relationships (call edges, references). - -**Timing:** -- <500 files: ~5-15 seconds -- 1,000-5,000 files: ~30-120 seconds -- 5,000+ files: use `--max-files 3000` to prevent timeout - -**Output:** `{status:"ok", files_scanned{html,css,python,...}, frontend{classes,ids}, backend{nodes,edges}}` - -**What if it's slow?** Use `--max-files N` to limit the number of files scanned. Use `--incremental` after the first scan to only rescan changed files (~1-5 seconds). - -**What if tree-sitter is not installed?** You see `WARNING: TSBackendParser init failed, using JS fallback: No module named 'tree_sitter'`. This is non-fatal — regex fallback kicks in automatically. +Any valid JSON response means setup is complete. -### Step 4: Verify the Setup +### Step 4: After Code Changes ```bash -$CLI query "main" /path/to/project +codelens scan /path/to/project --incremental ``` -**Expected:** `{status:"ok", found:true|false, ...}` — if you get a valid JSON response, your setup is complete. - -### Step 5: After Code Changes - -After modifying any source file, always run: - -```bash -$CLI scan --incremental /path/to/project -``` - -This rescans only changed files (~1-5 seconds). Without this, queries may return stale data. +Without this, queries return stale data. ### Common First-Time Issues | Issue | Cause | Fix | -|-------|-------|-----| -| `WARNING: TSBackendParser init failed` | tree-sitter not installed | Run `setup.sh`, or ignore (regex fallback works) | +|---|---|---| +| `WARNING: TSBackendParser init failed` | tree-sitter not installed | `bash setup.sh`, or ignore (regex fallback works) | | `Auto-detected workspace: ...` | Invalid workspace arg | Check the returned `workspace` field matches your project | +| Search silently returns 0 results | Argument order backwards | `search` takes `pattern` first, `workspace` second — every other command is the reverse | | Empty results after scan | No recognized source files | Check `.codelens/codelens.config.json` ignore list | -| `status: "error"` on any command | Registry not built | Run `init` then `scan` first | +| `status: "error"` on any command | Registry not built | Run `scan` first | | Scan takes too long | Very large repo | Use `--max-files 3000` | --- ## Workspace Auto-Detect -The `workspace` argument is **optional** for ALL commands. If omitted, CodeLens auto-detects via: - -1. Current directory (if has project markers: package.json, pyproject.toml, Cargo.toml, etc.) -2. Parent directories (walk up to 10 levels to find project root) -3. Last used workspace (cached at `~/.codelens/.codelens_last_workspace`) -4. Fallback: current working directory +The `workspace` argument is optional for every command: ```bash -$CLI scan # Auto-detect → works! -$CLI query "myFunc" # Auto-detect → works! -$CLI smell # Auto-detect → works! +codelens scan # Auto-detect → works! +codelens search "myFunc" --mode symbol # Auto-detect → works! +codelens audit --check smell # Auto-detect → works! ``` +Resolution order: current directory (project markers: package.json, pyproject.toml, Cargo.toml, ...) → parent directories (up to 10 levels) → last used workspace cache → current working directory. + --- ## AI Workflows -### Pre-Write Check (MANDATORY) +### Pre-Write Check (recommended) ``` -1. Check registry exists → if not: init + scan -2. query "name" → found: false = SAFE, active = EXTEND, dead = ASK, collision = STOP -3. Write code -4. scan --incremental +1. search "name" --mode symbol → not found = SAFE, active = EXTEND, dead = ASK +2. Write code +3. scan --incremental ``` ### Security Audit Chain ``` -secrets → dataflow (user_input→sinks) → env-check → vuln-scan +security --check secrets → security --check taint → security --check vuln-scan ``` ### Quality Gate ``` -smell → complexity → debug-leak → dead-code → a11y → secrets -``` - -### Pre-Deploy Checklist - -``` -secrets → debug-leak → env-check → config-drift → vuln-scan → dead-code +audit --check smell → audit --check complexity → audit --check dead-code → security --check secrets ``` ### Code Review ``` -scan --incremental → diff → list --filter dead → list --filter collision → missing-refs → secrets --severity critical +scan --incremental → deps --check circular → audit --check dead-code → security --check secrets ``` ### Bug Investigation ``` -search "pattern" → context "name" → trace --direction up → missing-refs +search "pattern" . --mode regex → context . --check trace --name X --direction up ``` ### New Feature Development ``` -query "name" → context (if exists) → side-effect → write → scan --incremental → missing-refs → test-map +search "name" --mode symbol → context (if exists) → audit --check side-effect → write → scan --incremental ``` --- @@ -218,54 +218,46 @@ query "name" → context (if exists) → side-effect → write → scan --increm ## Error Recovery | Failure | Recovery | -|---------|----------| +|---|---| | `scan` file read error | Skip unreadable files, scan the rest | | `scan` grammar import error | Fallback to regex parser automatically | -| `query` registry not found | Returns `found:false` (not an error) — run `init` + `scan` | -| `query` registry corrupt | Delete `.codelens/` → `init` → `scan` → retry | -| `trace` symbol not found | Try `search` first to locate, then `trace` with exact name | +| `search --mode symbol` not found | Returns `found:false`-equivalent (not an error) — run `scan` first if registry is missing | +| Registry corrupt | Delete `.codelens/` → `scan` → retry | +| `context --check trace` symbol not found | Try `search --mode symbol` first to locate the exact name | | `impact` no edges | Run `scan` first to build edges, then retry | -| `vuln-scan` no lockfile | Returns empty results — not an error | -| `ownership` no git repo | Fallback to mtime-based analysis | -| `perf-hint` too many results | Apply `--severity critical` or `--category` filter | -| Any command timeout | Use `--max-files` to reduce scope, or `--timeout` to increase budget | -| `ask` timeout (45s) | `status:"timeout"` — run the specific command directly | -| `analyze` engine timeout | `skipped:true` per-engine — run that command individually | -| `summary` budget exceeded | `timed_out_engines[]` — use `--detail minimal` or specific commands | -| `handbook` budget exceeded | `partial:true` — run individual commands for skipped sections | -| Any `status:"error"` | Follow the `suggestion` field in the error response | +| `security --check vuln-scan` no lockfile | Returns empty results — not an error | +| `history --check ownership` no git repo | Fallback to mtime-based analysis | +| Any command timeout | Use `--max-files` to reduce scope | +| Any `status:"error"` | Follow the `error`/`suggestion` field in the response | --- ## Status & Flag Reference | Status | Meaning | AI Action | -|--------|---------|-----------| +|---|---|---| | `active` | Used, ref_count > 0 | Normal, proceed | -| `dead` | Nothing references it | Flag to user | +| `dead` | Nothing references it in the graph | Cross-check with `trace --direction up` before flagging to user | | `duplicate_ref` | Referenced from many places | List all callers | -| `collision` | ID on >1 HTML element (bug) | STOP, fix first | -| `duplicate_define` | Defined >1x | Warning | +| `collision` | Same id/name defined ambiguously | Stop, fix first | +| `duplicate_define` | Defined more than once | Warning | -**Priority order:** collision → duplicate_define → dead → duplicate_ref → active → found:false +**Priority order:** collision → duplicate_define → dead → duplicate_ref → active → not found --- ## Reading the Output — Signal vs. Metric | Metric | What it actually means | How to interpret | -|--------|------------------------|------------------| +|---|---|---| | `reference_count` / caller count | **Popularity** — how often a symbol is referenced | Not a criticality signal. A payment-flow function called once is more critical than a utility called 50×. | -| `status: dead` | Nothing references it | Flag for removal — but verify it's not an entry point (HTTP handler, CLI subcommand, exported API). | -| `status: duplicate_ref` | Referenced from many places | List all callers with `trace --direction up` before changing. | +| `status: dead` | Nothing references it in the graph | Flag for removal — but verify it's not an entry point (HTTP handler, CLI subcommand, exported API) first. | +| `status: duplicate_ref` | Referenced from many places | List all callers with `context --check trace --direction up` before changing. | | `high_complexity` | Cyclomatic complexity ≥ threshold | Hotspot for bugs, not necessarily important. Cross-reference with `trace --direction up`. | -**To judge importance:** run `trace --direction up ` to see **who** calls it, then weigh by context (payment, auth, hot path) — not by raw count. +**To judge importance:** `context --check trace --name X --direction up` to see **who** calls it, then weigh by context (payment, auth, entry point) — not by raw count. -**To reduce noise:** -- `--format compact` — token-efficient single-char keys (AI/script consumption) -- `--lite` — minimal output (decision-making mode, per-command tailored) -- `--detail minimal` (summary) — critical-severity findings only +**To reduce noise:** `--format compact` for token-efficient output, `--lite` for decision-making mode, `--detail minimal` (where supported) for critical-severity findings only. **First scan is slow by design** — it builds the SQLite graph. Subsequent scans are incremental (`--incremental`). @@ -273,89 +265,61 @@ query "name" → context (if exists) → side-effect → write → scan --increm ## Integration with AI Agent -### CLI Integration (Recommended) +### CLI Integration ```python import subprocess, json -CLI = "/path/to/codelens/scripts/codelens.py" -def cl_query(name, workspace): - r = subprocess.run(["python3", CLI, "query", name, workspace], - capture_output=True, text=True, timeout=30) +def cl_search(name, workspace, mode="symbol"): + r = subprocess.run( + ["codelens", "search", name, workspace, "--mode", mode, "--lite"], + capture_output=True, text=True, timeout=30, + ) return json.loads(r.stdout) ``` ### Mandatory Rules -1. **Query before write** — ALWAYS call `query` before creating new class/id/function -2. **Scan after write** — Run `scan --incremental` after modifying code -3. **STOP on collision** — Do not proceed if ID collision detected -4. **Report dead code** — Show it to user, don't silently ignore -5. **Handle errors** — Gracefully handle subprocess timeouts and JSON parse errors +1. **Search before write** — always check for an existing symbol before creating a new class/id/function +2. **Scan after write** — run `scan --incremental` after modifying code +3. **Report dead code, don't silently ignore it** +4. **Handle errors** — gracefully handle subprocess timeouts and JSON parse errors ### Token Budget Strategy -CodeLens has **smart defaults** that prevent token overflow without any flags: - -1. **Auto `--top 20`** — List commands automatically limit to 20 items. No configuration needed. -2. **Sort-aware truncation** — `--top N` sorts by relevance first (severity, complexity, etc.), so you always get the most important items. -3. **`--lite`** for `query` — returns just `{found, action}` instead of full node details + callers + callees -4. **`--max-tokens 500`** for strict budgets — automatically truncates largest lists to fit -5. **`--format ai`** — normalizes output to consistent `{stats, items[], truncated, recommendations}` schema -6. **`--severity critical`** for `smell`, `secrets`, `perf-hint`, `vuln-scan` — filters noise -7. **`--category`** filters on `dead-code`, `smell`, `perf-hint`, `debug-leak` — narrow scope -8. Use `query --lite` before `context` — if `found:false`, no need for context -9. Use `--top 0` to override smart defaults and get unlimited results -10. Set `CODELENS_AI_MODE=1` env var to make `--format ai` the default output format -11. Auto-setup caps at 3000 files — run `scan` manually for full analysis on large repos - -### Auto-Setup Behavior - -When no `.codelens/` registry exists, any analysis command auto-runs `init` + `scan`. This is transparent — you don't need to think about setup. - -**Timeout protection**: Auto-setup caps scanning at **3000 files** to prevent long waits. If your repo has more files, the auto-setup will be fast but partial. For full analysis, run `scan` manually: - -```bash -$CLI scan # Full scan (no file limit) -``` - -The `_auto_setup` field in the response tells you if it was capped: -```json -{"auto_setup": true, "capped": true, "hint": "Auto-setup capped at 3000 files. Run 'scan' manually for full analysis."} -``` +1. **`--top N`** — sorts by relevance first (severity, complexity, ...), then truncates +2. **`--lite`** — command-specific minimal payload (see table above) +3. **`--max-tokens N`** — hard cap, truncates the largest lists to fit +4. **`--format ai`** — normalizes output to `{stats, items[], truncated, recommendations}` +5. **`--format compact`** — single-char keys, smallest payload +6. Set `CODELENS_AI_MODE=1` to make `--format ai` the default output format +7. Auto-setup caps scanning at 3000 files — run `scan` manually (no cap) for full analysis on large repos ### CODELENS_AI_MODE -Set the `CODELENS_AI_MODE` environment variable to `1`, `true`, or `yes` to make `--format ai` the **default** output format. This eliminates the need to add `--format ai` to every command. - ```bash export CODELENS_AI_MODE=1 -$CLI smell # Now outputs in --format ai by default -$CLI complexity # Same +codelens audit --check smell # now outputs in --format ai by default ``` -Without this env var, the default format is `json` (backward compatible). +Without this env var, the default format is `json`. ### Reference Files -- `references/agent-integration.md` — Full integration guide (CLI, Python API, JSON schemas, decision trees) -- `references/parser-rules.md` — Parsing rules per language -- `references/query-examples.md` — Query examples and output interpretation -- `references/status-codes.md` — Details for all statuses and flags +- [docs/agent-usage-guide.md](docs/agent-usage-guide.md) — verified per-language coverage, `--lite` reducer coverage, known gaps, with real before/after fix evidence +- `references/agent-integration.md` — CLI/Python API integration guide +- `references/parser-rules.md` — parsing rules per language +- `references/query-examples.md` — query examples and output interpretation +- `references/status-codes.md` — details for all statuses and flags --- -## v8.x Feature Summary - -CodeLens v8.0+ adds 7 major capability pillars over v7.x: - -1. **AST Taint Engine** (`taint` command) — Tree-sitter AST traversal, path-sensitive, scope-aware, inter-procedural taint tracking with confidence scoring and taint path rendering. Default engine when tree-sitter is available. -2. **Live CVE/OSV Scanning** (`vuln-scan` v2) — Real-time data from OSV.dev API across 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) with SQLite cache + offline fallback. -3. **Plugin System** (`plugin` command) — 4 plugin types (rule_pack / engine / formatter / command), 3-tier discovery (local > user > built-in). Ships with OWASP Top 10 (36 rules) + Compliance (53 rules: PCI-DSS v4.0 + HIPAA). -4. **VS Code Extension** (`vscode-codelens/`) — Diagnostics on save, QuickFix code actions, guard pre-save hooks, status bar health indicator, SARIF v2.1.0 integration. -5. **Cross-File Dataflow Engine** (`dataflow` v2) — Workspace-wide call graph with import resolution (`from/import`, `require` destructuring) and bidirectional taint propagation. -6. **OWASP Top 10 + Compliance Mapping** — 89 rules total (A01-A10 + PCI-DSS requirements 1-12 + HIPAA 45 CFR § 164.312). -7. **CI/CD Quality Gate** (`check` command) — Exits non-zero on failure, SARIF output for GitHub Advanced Security / VS Code. -8. **Gitleaks-Backed Secrets Scanner** (`secrets` command, issue #159) — When [gitleaks](https://github.com/gitleaks/gitleaks) is installed, `codelens secrets` uses it as the primary backend for 600+ maintained rules and entropy scoring. Falls back to the built-in regex scanner when gitleaks is unavailable (opt-in upgrade, never a hard dependency). Use `--no-gitleaks` to force the regex backend. Install: `brew install gitleaks` / `go install github.com/gitleaks/gitleaks/v8@latest` / [GitHub releases](https://github.com/gitleaks/gitleaks/releases). +## Feature Summary -v8.1 follows up with F1 benchmark improvements (avg F1 0.803 → 0.872), circular engine depth fixes (F1 0.667 → 1.000), dead-code engine fixes (F1 0.800 → 0.952), and AST taint depth enhancements (return-value propagation, scope-hierarchical TaintState, branch condition refinement). +- **AST Taint Engine** (`security --check taint`) — tree-sitter AST traversal, path-sensitive, scope-aware, inter-procedural taint tracking with confidence scoring. **Python/JS/TS/TSX only** — no Rust source/sink rules yet. +- **Live CVE/OSV Scanning** (`security --check vuln-scan`) — real-time data from OSV.dev across 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex), SQLite cache + offline fallback. +- **Plugin System** (`plugin` command) — 4 plugin types (rule_pack/engine/formatter/command), 3-tier discovery. Ships with OWASP Top 10 (36 rules) + Compliance (53 rules: PCI-DSS v4.0 + HIPAA). +- **Cross-File Dataflow Engine** (`impact --check dataflow`) — workspace-wide call graph with import resolution and bidirectional taint propagation. +- **CI/CD Quality Gate** (`check` command) — exits non-zero on failure, SARIF output for GitHub Advanced Security / VS Code. +- **Gitleaks-Backed Secrets Scanner** (`security --check secrets`) — uses [gitleaks](https://github.com/gitleaks/gitleaks) as the primary backend when installed (600+ rules, entropy scoring), falls back to the built-in regex scanner otherwise. `--no-gitleaks` forces the regex backend. +- **Rust-aware dead-code detection** — `#[cfg(test)] mod tests { #[test] fn ... }` inline test functions are correctly exempted (fixed 2026-07-12; previously 56%+ of Rust `registry_dead` findings were test-function false positives). diff --git a/scripts/codelens.py b/scripts/codelens.py index 3f39867..3b6318a 100755 --- a/scripts/codelens.py +++ b/scripts/codelens.py @@ -1018,7 +1018,7 @@ def main(): description=( f"CodeLens v{CODELENS_VERSION} — Live Codebase Reference Intelligence " f"(Tree-sitter Edition). {_command_count} commands available; run " - f"`python3 scripts/codelens.py --command-count` to print just the count." + f"`codelens --command-count` to print just the count." ) ) # Quick introspection flag — prints the runtime command count and exits.