diff --git a/docs/agent-usage-guide.md b/docs/agent-usage-guide.md new file mode 100644 index 00000000..cdac7cea --- /dev/null +++ b/docs/agent-usage-guide.md @@ -0,0 +1,219 @@ +# CodeLens as an Agent's Primary Code-Intelligence Tool + +> **Status:** Living reference, verified against a real 425-file polyglot +> workspace (rs/ts/tsx/js/css/html + Python for CodeLens itself). +> **Last verified:** 2026-07-12 +> **Purpose:** Tell an agent what to reach for instead of manual `grep`, and +> what NOT to trust yet. + +This is not API documentation (see `--help` per command for that). This is +the accumulated result of exercising every umbrella command end-to-end on a +real Tauri + React + Rust codebase and fixing what broke. + +--- + +## The one rule that will burn you + +**`search` takes `pattern` first, `workspace` second** — every other umbrella +command (`audit`, `deps`, `context`, `security`, ...) takes `workspace` +first. Getting this backwards does **not** error — it silently searches for +the workspace path as the pattern and returns an empty `"ok"` result. If a +search comes back suspiciously empty, check argument order before assuming +the symbol doesn't exist. + +``` +codelens search "pattern" --mode symbol # correct +codelens audit --check dead-code # different order, also correct +``` + +--- + +## Replacing grep: which mode for which question + +| Question | Use | Notes | +|---|---|---| +| "Where is symbol X defined?" | `search "X" . --mode symbol` | Exact name match across all languages in one call. | +| "What calls/is called by X?" | `context . --check trace --name X --direction up\|down` | Full transitive chain with depth, crosses file *and* language boundaries (verified: TS → Rust in one chain). | +| "Find code related to concept Y" (fuzzy) | `search "Y" . --mode semantic` | TF-IDF over symbol names/paths, not full-text — good for "where's the auth code", not literal string matches. | +| "Find this exact string/regex" | `search "regex" . --mode regex --type ts` | This is the real grep replacement. **Always pass `--type`** (html/css/js/ts/tsx/rust/python/vue/svelte) — without it, the default result cap can silently truncate the walk (see "max-results early exit" below) before reaching your target file if match density is skewed toward certain paths. | +| "Structural/graph question" (e.g. "all functions calling any DB write") | `search "MATCH (f)-[:CALLS]->(g:function) WHERE ..." . --mode graph` | Cypher subset — replaces chaining trace+impact+context by hand. | +| "Is this safe to delete?" | `audit . --check dead-code` **plus** `context . --check trace --name X --direction up` | Don't trust dead-code alone — cross-check with trace. See Known Gaps. | +| "What imports this file?" | `deps . --check dependents --files path/to/file.ts` | | +| "Any circular imports?" | `deps . --check circular` | | +| "Any secrets/vulnerable deps/injection risk?" | `security . --check secrets\|vuln-scan\|taint\|regex-audit` | **Taint is Python/JS/TS/TSX only** — see Known Gaps, no Rust coverage. | +| "10-second repo orientation" | `context . --check orient` (or bare `context .`, it's the default) | Framework detection, dev commands, entry points. | +| "Prioritized health snapshot" | `summary .` | Aggregates dead-code/smell/taint/vuln-scan; use `--lite` for an agent-sized payload. | + +--- + +## `--lite`: use it, but know its coverage + +`--lite` is the actual token-budget lever for agent use — full non-lite +output on a real workspace routinely runs 10-50x larger. As of this session +it works correctly for **all 12 umbrella commands** (was previously broken +for every umbrella — see Fixed This Session). Coverage of *dedicated* +reducers (extra compression beyond the generic fallback): + +- **Rich, hand-tuned:** `query`, `impact`, `smell`, `complexity`, `dead-code`, + `debug-leak`, `perf-hint`, `secrets`, `taint`, `a11y`/`css-deep`/ + `regex-audit`/`vuln-scan`, `summary`, `history`. +- **Generic fallback (adequate, not hand-tuned):** everything else — + `orient`, `outline`, `trace`, `context`, `api-map`, `doctor`, `circular`, + `affected`, `dependents`. Still bounded (carries scalar fields + first 5 + of the primary list), just not as surgically trimmed. + +If a `--lite` result on some command looks emptier than it should, check +whether it hits a dedicated reducer or the generic fallback — the generic +fallback only knows a fixed set of top-level key names +(`found`/`action`/`risk`/`health_score`/`query`/`symbol`/`workspace` + +`stats`/`recommendations` + one primary list). Deeply nested fields it +doesn't recognize get silently dropped, not an error. + +--- + +## Per-language coverage (verified) + +| Language | scan/parse | search | trace/context | audit dead-code | security taint | security secrets/vuln-scan | +|---|---|---|---|---|---|---| +| TypeScript/TSX | ✓ (`.ts` counted under the `tsx` scan-stat bucket, cosmetic only) | ✓ | ✓ (verified 28-caller chain) | ✓ | ✓ | ✓ | +| JavaScript | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| React (JSX/TSX components) | ✓ | ✓ | ✓ | ✓ (correctly distinguishes "default export unused" from "named export used" — verified against a real component) | ✓ | ✓ | +| Python | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Rust | ✓ | ✓ | ✓ (trace chains cross into `.rs` from TS/TSX call sites) | ✓ **after this session's fix** — inline `#[cfg(test)] mod tests { #[test] fn ... }` no longer false-positives (was ~56%+ of registry_dead noise before) | **✗ not supported** — engine is Python/JS/TS/TSX only, no Rust source/sink rules | ✓ (regex/gitleaks are language-agnostic) | +| CSS | ✓ (class/id extraction) | ✓ regex mode | n/a (no call graph for CSS) | n/a | n/a | ✓ (regex/gitleaks scan text) | +| HTML | ✓ | ✓ regex mode | n/a | n/a | n/a | ✓ | + +**Rust dead-code residual gap (issue #228, not fixed this session):** `impl` +blocks and trait-default methods are still structurally false-flagged as +dead in some cases — this is a different, deeper false-positive source than +the test-function one fixed above (parser doesn't yet understand a trait impl +satisfies a contract rather than being "called"). Cross-check any Rust +`impl`-typed dead-code finding with `trace --direction up` before trusting it. + +--- + +## Fixed this session (verified, not just claimed) + +All fixes verified by direct CLI reproduction + `pytest` (added regression +tests where the bug wasn't already covered) before/after comparison. No +worker involved per explicit user directive — root cause identified by +reading the actual failure, not pattern-matched from symptoms. + +1. **`search --help` example order was backwards** (`scripts/commands/search.py`) + — the epilog/docstring examples showed `workspace` before `pattern`, + opposite of the actual argparse signature. Following the documented + example silently returns an empty result. Fixed the docstring/epilog; + the actual argument order was already correct. + +2. **`graph` mode `truncated` flag was meaningless** (`query_graph_engine.py`) + — set to `True` whenever the query merely *contained* a `LIMIT` clause, + regardless of whether any rows were actually cut off. A `LIMIT 50` query + matching 1 row reported `truncated: true`, misleading callers into + thinking more results existed. Now computed from actual row count vs. + limit. Added `tests/test_query_graph.py::test_truncated_flag_false_when_limit_exceeds_match_count`. + +3. **`--lite` was completely broken for all 12 umbrella commands** + (`scripts/codelens.py::_apply_lite`) — the reducer dispatch table + predates the #195 umbrella consolidation and keyed off the *old* leaf + command names (`smell`, `dead-code`, `query`, ...). Since umbrella + commands always pass their own name (`context`, `audit`, `security`, + ...), no branch ever matched — every umbrella's `--lite` output + silently collapsed to `{"status": "ok"}` with all data dropped. Fixed + by unwrapping the `{"s","st","r":[{"_check":name,...}]}` envelope, + applying the existing per-check reducers keyed by each item's own + `_check` name, then re-wrapping. Verified: `audit --check dead-code + --lite` now returns `removal_safety`/`stats`/`top_items` as designed. + +4. **Rust inline test functions false-flagged as dead code** + (`deadcode_engine.py::_detect_dead_from_registry`) — the existing + test-file exemption only matched separate-directory conventions + (`/tests/`, `/__tests__/`), which doesn't cover Rust's idiomatic + `#[cfg(test)] mod tests { #[test] fn ... }` living inline in the same + file as production code. On the verified workspace this was **56%+ of + all Rust `registry_dead` findings** (real measurement: 100→69 in the + top-100 window after the fix, with `mod tests` blocks also newly + exempted). Fixed by peeking at the source lines above a flagged + symbol for a `#[test]`-family attribute. Added + `tests/test_deadcode_engine.py::test_registry_dead_exempts_rust_inline_test_functions`. + +5. **`deps --check import-snapshot` was permanently non-functional** + (issue #218) — `export-snapshot` was dropped entirely in #195 with no + replacement, so `import-snapshot` could never find a snapshot file to + load; the underlying `build_snapshot()`/`write_snapshot()` logic in + `snapshot_io.py` was never deleted, just orphaned. Added + `scripts/commands/export_snapshot.py` (new `deps --check + export-snapshot` sub-mode) and verified a full export→import round + trip preserves node/edge counts. Also excluded both snapshot checks + from the bare `codelens deps ` default (they're + side-effecting/opt-in and previously always showed a spurious error + with no `--input`/`--output` given). See + `docs/design/0218-export-snapshot.md` and + `tests/test_export_snapshot.py`. + +6. **`summary --lite` didn't actually reduce output** — summary's own + `--help` describes it as "anti-overload prioritized findings", but + `--lite` fell through to the generic fallback which only trims the + *outer* `findings` list, not each finding's nested `top_items` (and + dataflow findings nest a full `flow_chain` per item). On the verified + workspace, `summary . --lite` returned the same multi-thousand-token + payload as the non-lite call. Added a dedicated `summary` reducer that + trims each category's `top_items` to 3 and strips `flow_chain`. Added + regression test. + +7. **`history --lite` collapsed to `{"status", "workspace"}`** — same root + cause as #6: history's real payload (`snapshots`, `latest`, `trends`, + `deltas`) lives under keys the generic fallback doesn't recognize. + Added a dedicated `history` reducer. Added regression test. + +8. **Windows CRLF corruption in all stdout/stderr output** — `_force_utf8_stdio()` + (the fix for issue #179's Unicode arrow crash) re-wraps stdout/stderr + with `io.TextIOWrapper(..., encoding='utf-8')` but never set `newline=''`, + so every `\n` written on Windows silently became `\r\n` (Python's + default write-side `os.linesep` translation). This affects every JSON/ + text line CodeLens prints on Windows — harmless for JSON *parsing* but + breaks byte-exact comparisons and any tool assuming Unix line endings. + Fixed by adding `newline=''`. This was already caught by an existing + test (`test_writes_unicode_arrow_to_replaced_stream`) that was failing + before this fix. + +## Test suite baseline (verified 2026-07-12, post-fixes) + +Full suite (`pytest tests/ --ignore=test_integration.py --ignore=test_lsp_server.py +--ignore=test_large_file_parsing.py`) has ~20 pre-existing failures unrelated +to this session's changes — confirmed by isolating each one and checking it +touches files never edited today (Windows path-separator assumptions in +`test_codelensignore.py`/`test_compact_format.py`/`test_history_engine.py`/ +`test_secrets_gitleaks.py`, `os.geteuid()` not existing on Windows in +`test_doctor.py`, schema-version test debt in `test_confidence.py`, etc.). +Two additional failures observed this session +(`test_cli.py::TestArgparseFormatConflictRegression::test_scan_with_format_*`) +are pure environment slowness, not a functional break — `codelens scan .` on +the CodeLens repo itself (400+ Python files) exceeded the test's 60s +subprocess timeout before the actual assertion (no argparse error in stderr) +was ever reached. None of these block real usage; they're Windows-vs-POSIX +test debt, not product bugs. + +## Known limitations (not fixed — scope, not a quick bug) + +- **No Rust taint analysis.** `security --check taint` only analyzes + Python/JS/TS/TSX (`ast_taint_engine.get_supported_languages()`). A Tauri + app that shells out via `std::process::Command` (verified: this workspace + has several `Command::new(...)` sinks fed by `std::env::var()` sources in + `.rs` files) gets zero taint coverage on the Rust side. This is a real + feature gap for `harus berkerja di rs` — building Rust source/sink rules + + AST walking is a multi-day feature, not a bug fix, so it wasn't + attempted this session. Tracked as a GitHub issue for follow-up. +- **Rust `impl`-block dead-code false positives** (issue #228) — separate, + deeper false-positive source than the test-function one fixed this + session. Still open. +- **`.ts` files are silently counted under the `tsx` bucket** in `scan` + output stats (`files_scanned.tsx` = actual `.tsx` + `.ts` count). Cosmetic + only — parsing/analysis is correct, just the reported stat label is + misleading if you're trying to verify TS vs TSX file counts from `scan` + output alone. +- **`language` field is always empty (`""`) in `search --mode semantic` + results** — `graph_nodes` (the table semantic search reads from) doesn't + carry a language column; documented as intentional in + `semantic_search_engine.py`'s own docstring (file extension already + conveys language). Not fixed — would need a schema change for + low value (extension already tells you the language). diff --git a/docs/design/0218-export-snapshot.md b/docs/design/0218-export-snapshot.md new file mode 100644 index 00000000..ff5285fa --- /dev/null +++ b/docs/design/0218-export-snapshot.md @@ -0,0 +1,67 @@ +# Design Doc: export-snapshot (restore deps import-snapshot) + +> **Status:** Accepted +> **Date:** 2026-07-12 +> **Author:** Claude (direct fix, no worker — user directive) +> **Related issues:** #218 + +--- + +## Problem + +`deps --check import-snapshot` reads `.codelens/snapshot.codelens.gz` and loads +it into the graph DB, but no command produced that file: `export-snapshot` was +one of the commands dropped entirely in the #195 umbrella consolidation, while +`import-snapshot` survived as a `deps` sub-check. Every `import-snapshot` run +on a workspace that never had the old standalone `export-snapshot` command +run fails with `"Snapshot file not found"` — the feature was permanently +non-functional. `snapshot_io.py`'s `build_snapshot()`/`write_snapshot()` +(the actual export logic) were never deleted, only the CLI entry point that +called them. + +Separately: the bare `codelens deps ` (no `--check`) ran every +registered check including `import-snapshot`, which always failed with no +`--input` given — every default `deps` run showed a spurious error entry +unrelated to what the caller asked for. + +## Goal + +`codelens deps --check export-snapshot` writes a snapshot that +`codelens deps --check import-snapshot` can load back, restoring +the same node/edge counts (round trip). The bare `codelens deps ` +default only runs the read-only analyses (affected/dependents/circular). + +## Changes + +### New Files +- `scripts/commands/export_snapshot.py` — thin CLI wrapper around the + existing `snapshot_io.build_snapshot()` / `write_snapshot()`, mirroring + `import_snapshot.py`'s structure (same error-handling shape, same + `status`/`error` result contract). + +### Modified Files +- `scripts/commands/deps.py` — registered `export-snapshot` in `_CHECKS`; + added `--output` flag; added `_DEFAULT_EXCLUDED_CHECKS` so + `import-snapshot`/`export-snapshot` (side-effecting, opt-in) are excluded + from the bare `codelens deps ` "run everything" default. + +### Not Changed +- `snapshot_io.py` — export logic already existed and needed no changes. +- No new top-level CLI command — `export-snapshot` is a `deps --check` + sub-mode only, consistent with how `import-snapshot` itself is exposed + post-#195 (confirmed via `tests/test_issue195_consolidation.py`, which + still asserts `export-snapshot` is not a *standalone* command). + +## Testing + +`tests/test_export_snapshot.py`: export creates a valid `.gz`, missing-DB +error path, custom `--output` path, full export→import round trip (node/edge +counts preserved into a fresh workspace), and the default-check-list +exclusion for both snapshot checks. + +## Alternatives Considered + +- Re-adding `export-snapshot` as a standalone top-level command: rejected — + contradicts the #195 consolidation (12 umbrellas only) and `import-snapshot` + itself already lives as a `deps` sub-check, so symmetry argues for the same + placement. diff --git a/scripts/codelens.py b/scripts/codelens.py index 4d343d90..3f398678 100755 --- a/scripts/codelens.py +++ b/scripts/codelens.py @@ -124,7 +124,15 @@ def _force_utf8_stdio() -> None: setattr( sys, _name, - io.TextIOWrapper(_buffer, encoding='utf-8', errors='replace'), + # newline='' disables TextIOWrapper's default write-side + # translation of '\n' to os.linesep. Without it, every '\n' in + # JSON/text output becomes '\r\n' on Windows even though the + # source strings only ever contain '\n' — silently breaking + # byte-exact comparisons and any downstream tool that assumes + # Unix line endings (matches the already-UTF-8 no-op path + # above, which never translates on Linux/macOS since + # os.linesep is '\n' there). + io.TextIOWrapper(_buffer, encoding='utf-8', errors='replace', newline=''), ) @@ -657,7 +665,36 @@ def _apply_max_tokens(result: Dict[str, Any], max_tokens: int) -> Dict[str, Any] # ─── Post-Processing: --lite ─────────────────────────────────── def _apply_lite(result: Dict[str, Any], command: str) -> Dict[str, Any]: - """Reduce output to minimum viable for AI decision-making.""" + """Reduce output to minimum viable for AI decision-making. + + Umbrella commands (issue #195) wrap sub-analysis output as + ``{"s":.., "st":.., "r":[{"_check": name, ...}, ...]}``. The per-check + reducers below (``_apply_lite_single``) were written against the old + pre-umbrella leaf commands (``smell``, ``dead-code``, ``query``, ...) + and keyed off the top-level command name. Since #195/#199/#200, + ``command`` is always one of the 12 umbrella names (``context``, + ``audit``, ``security``, ...) which never match any branch below — + every umbrella command's --lite output silently collapsed to + ``{"status": "ok"}`` with all data dropped. Fix: unwrap the envelope, + apply the existing per-check reducer to each item keyed by its own + ``_check`` name, then re-wrap. + """ + if isinstance(result, dict) and isinstance(result.get("r"), list) and "_check" in ( + result["r"][0] if result["r"] and isinstance(result["r"][0], dict) else {} + ): + lite_items = [ + _apply_lite_single(item, item.get("_check", command)) if isinstance(item, dict) else item + for item in result["r"] + ] + for lite_item, orig_item in zip(lite_items, result["r"]): + if isinstance(lite_item, dict) and isinstance(orig_item, dict) and "_check" in orig_item: + lite_item["_check"] = orig_item["_check"] + return {"s": result.get("s", "ok"), "st": result.get("st"), "r": lite_items} + return _apply_lite_single(result, command) + + +def _apply_lite_single(result: Dict[str, Any], command: str) -> Dict[str, Any]: + """Reduce a single (non-enveloped) sub-result to minimum viable output.""" if not isinstance(result, dict): return result @@ -679,6 +716,70 @@ def _apply_lite(result: Dict[str, Any], command: str) -> Dict[str, Any]: "action": result.get("recommended_action") or result.get("action"), } + if command == "history": + # Same class of bug as "summary" above: history's real payload + # (snapshots count, latest snapshot's health metrics, trends, + # deltas) lives under keys the generic fallback doesn't know about + # (only checks a fixed scalar-key allowlist plus a couple of + # well-known list keys), so --lite collapsed to just + # {"status", "workspace"} with zero actual history data. + lite = { + "status": result.get("status", "ok"), + "workspace": result.get("workspace"), + "snapshots": result.get("snapshots"), + } + latest = result.get("latest") + if isinstance(latest, dict): + lite["latest"] = { + k: latest[k] for k in ( + "timestamp", "health_score", "total_findings", + "findings_by_severity", "avg_complexity", + "high_complexity_count", + ) if k in latest + } + if result.get("trends"): + lite["trends"] = result["trends"] + if result.get("deltas"): + lite["deltas"] = result["deltas"] + return lite + + if command == "summary": + # Summary's own job is "anti-overload prioritized findings" (its + # --help description), so --lite must actually be minimal. The + # generic fallback below only trims the outer `findings` list, not + # each finding's nested `top_items` (and dataflow findings nest a + # full flow_chain per item) — a --lite summary on a real workspace + # was coming back with thousands of tokens of untouched detail, + # defeating the point of the flag. + lite = { + "status": result.get("status", "ok"), + } + for key in ("workspace", "identity", "frameworks", "is_monorepo"): + if key in result: + lite[key] = result[key] + if result.get("recommendations"): + lite["recommendations"] = result["recommendations"][:3] + findings = result.get("findings", []) + if findings: + lite_findings = [] + for f in findings: + if not isinstance(f, dict): + continue + lf = {k: v for k, v in f.items() if k != "top_items"} + top_items = f.get("top_items") + if isinstance(top_items, list) and top_items: + trimmed = [] + for item in top_items[:3]: + if isinstance(item, dict) and "flow_chain" in item: + item = {k: v for k, v in item.items() if k != "flow_chain"} + trimmed.append(item) + lf["top_items"] = trimmed + if len(top_items) > 3: + lf["top_items_total"] = len(top_items) + lite_findings.append(lf) + lite["findings"] = lite_findings + return lite + if command == "smell": # Smell lite: health score + top 5 actionable items + action lite = { diff --git a/scripts/commands/deps.py b/scripts/commands/deps.py index 21f6fa79..3cf5ded5 100644 --- a/scripts/commands/deps.py +++ b/scripts/commands/deps.py @@ -50,8 +50,17 @@ "module": "commands.import_snapshot", "help": "Import a .codelens.gz snapshot into the graph DB", }, + "export-snapshot": { + "module": "commands.export_snapshot", + "help": "Export the graph DB to a .codelens.gz snapshot (issue #218)", + }, } +# Checks that are NOT run by default when --check is omitted (import/export +# are explicit opt-in actions with side effects — a bare `codelens deps .` +# should only run the read-only analyses). +_DEFAULT_EXCLUDED_CHECKS = {"import-snapshot", "export-snapshot"} + ALL_CHECKS = list(_CHECKS.keys()) @@ -64,12 +73,17 @@ def add_args(parser): " dependents Module-level import tracking\n" " circular Circular dependency detection\n" " import-snapshot Import .codelens.gz into graph DB\n" + " export-snapshot Export graph DB to .codelens.gz (issue #218)\n" "\n" "Examples:\n" - " codelens deps . # all checks\n" + " codelens deps . # affected/dependents/circular\n" " codelens deps . --check circular # only circular\n" " codelens deps . --check affected,dependents # pick subset\n" + " codelens deps . --check export-snapshot --output s.codelens.gz\n" " codelens deps . --check import-snapshot --input s.codelens.gz\n" + "\n" + "NOTE: import-snapshot/export-snapshot are NOT run by the bare\n" + "`codelens deps ` default (side-effecting, opt-in only).\n" ) parser.add_argument("workspace", nargs="?", default=None, help="Path to workspace root (auto-detected if omitted)") @@ -94,6 +108,8 @@ def add_args(parser): help="circular: max cycles per type") parser.add_argument("--input", default=None, help="import-snapshot: path to .codelens.gz file") + parser.add_argument("--output", default=None, + help="export-snapshot: path to write .codelens.gz file") parser.add_argument("--merge", action="store_true", default=False, help="import-snapshot: deduplicate with existing graph") parser.add_argument("--db-path", default=None, @@ -103,7 +119,13 @@ def add_args(parser): def _parse_checks(check_arg: str) -> List[str]: """Parse --check argument into a list of valid check names.""" if not check_arg: - return list(ALL_CHECKS) + # import-snapshot/export-snapshot are explicit opt-in actions with + # side effects (write to the graph DB / filesystem) and always fail + # by default (no snapshot file present yet) — excluded from the + # bare `codelens deps ` "run everything" default so it + # doesn't always show a spurious error entry. Still runnable via + # explicit --check import-snapshot / --check export-snapshot. + return [c for c in ALL_CHECKS if c not in _DEFAULT_EXCLUDED_CHECKS] parts = [c.strip() for c in check_arg.split(",") if c.strip()] invalid = [p for p in parts if p not in _CHECKS] if invalid: @@ -157,6 +179,9 @@ def _build_namespace(base_args, check_name: str) -> argparse.Namespace: ns.input = getattr(base_args, "input", None) ns.merge = getattr(base_args, "merge", False) ns.db_path = getattr(base_args, "db_path", None) + elif check_name == "export-snapshot": + ns.output = getattr(base_args, "output", None) + ns.db_path = getattr(base_args, "db_path", None) return ns diff --git a/scripts/commands/export_snapshot.py b/scripts/commands/export_snapshot.py new file mode 100644 index 00000000..8f2f01f7 --- /dev/null +++ b/scripts/commands/export_snapshot.py @@ -0,0 +1,117 @@ +"""Export-snapshot command — Save the CodeLens graph as a portable snapshot. + +Companion to ``import-snapshot`` (issue #218): writes the current +``.codelens/codelens.db`` graph tables to a gzip-compressed JSON snapshot +(``.codelens/snapshot.codelens.gz`` by default) that a teammate can load +with ``codelens deps --check import-snapshot`` without running +a full ``codelens scan`` themselves. + +The snapshot contains graph metadata only (paths, symbols, edges) — +never file content. + +Usage:: + + codelens deps [workspace] --check export-snapshot [--output path] +""" + +# @WHO: scripts/commands/export_snapshot.py +# @WHAT: Export the graph DB to a portable .codelens.gz snapshot (issue #218). +# @PART: commands +# @ENTRY: execute() + +import os +import sys +from typing import Any, Dict, Optional + +from commands import register_command +from utils import default_db_path, logger +from snapshot_io import ( + build_snapshot, + default_snapshot_path, + format_size, + write_snapshot, +) + + +def add_args(parser): + """Add export-snapshot arguments to the parser.""" + parser.add_argument("workspace", nargs="?", default=None, + help="Path to workspace root (auto-detected if omitted)") + parser.add_argument("--output", default=None, + help="Output path for the snapshot archive " + "(default: .codelens/snapshot.codelens.gz)") + parser.add_argument("--db-path", default=None, + help="Custom path for the source SQLite database file") + + +def execute(args, workspace): + """Execute the export-snapshot command.""" + output_path = getattr(args, "output", None) + db_path = getattr(args, "db_path", None) + return cmd_export_snapshot(workspace, output_path=output_path, db_path=db_path) + + +def cmd_export_snapshot( + workspace: str, + output_path: Optional[str] = None, + db_path: Optional[str] = None, +) -> Dict[str, Any]: + """Write the CodeLens SQLite graph tables to a snapshot archive. + + Args: + workspace: Path to the workspace root. + output_path: Optional explicit snapshot path. If None, defaults to + ``/.codelens/snapshot.codelens.gz``. + db_path: Optional source SQLite db path. Defaults to + ``/.codelens/codelens.db``. + + Returns: + Dict with keys: ``status``, ``message``, ``output_path``, + ``bytes_written``, ``header``, ``workspace``, ``db_path``. + On error: ``status="error"`` with an ``error`` message. + """ + workspace = os.path.abspath(workspace) + effective_db = db_path or default_db_path(workspace) + + effective_output = output_path if output_path and os.path.isabs(output_path) \ + else os.path.join(workspace, output_path) if output_path \ + else default_snapshot_path(workspace) + + try: + snapshot = build_snapshot(workspace, db_path=effective_db) + except FileNotFoundError as exc: + return { + "status": "error", + "error": str(exc), + "workspace": workspace, + } + except Exception as exc: + logger.error(f"export-snapshot: build failed: {exc}", exc_info=True) + return { + "status": "error", + "error": f"Failed to build snapshot: {exc}", + "workspace": workspace, + } + + try: + bytes_written = write_snapshot(snapshot, effective_output) + except OSError as exc: + return { + "status": "error", + "error": f"Failed to write snapshot: {exc}", + "workspace": workspace, + } + + header = snapshot.get("header", {}) + message = f"Snapshot exported: {effective_output} ({format_size(bytes_written)})" + print(message, file=sys.stderr) + + return { + "status": "ok", + "message": message, + "output_path": effective_output, + "bytes_written": bytes_written, + "header": header, + "workspace": workspace, + "db_path": effective_db, + } diff --git a/scripts/commands/search.py b/scripts/commands/search.py index b9cce18d..a1c52d02 100644 --- a/scripts/commands/search.py +++ b/scripts/commands/search.py @@ -8,10 +8,15 @@ Default mode is **semantic** (find symbols by meaning). Switch via --mode: - codelens search "google auth" # semantic - codelens search "google auth" --mode symbol # exact name - codelens search "google auth" --mode regex # regex code - codelens search "MATCH (n) WHERE n.id CONTAINS x" --mode graph + codelens search "google auth" # semantic + codelens search "google auth" --mode symbol # exact name + codelens search "google auth" --mode regex # regex code + codelens search "MATCH (n) WHERE n.id CONTAINS x" --mode graph + +NOTE: pattern comes FIRST, workspace SECOND — opposite of every other umbrella +command (audit/deps/context/security all take workspace first). Getting this +backwards does not error: the workspace path silently becomes the search +pattern and returns an empty "ok" result. For raw Cypher pass-through (power user), prefer ``codelens graph ``. @@ -40,10 +45,14 @@ def add_args(parser): " graph Cypher-subset graph query (MATCH/WHERE/RETURN/LIMIT)\n" "\n" "Examples:\n" - " codelens search . \"google auth\" # semantic (default)\n" - " codelens search . \"google auth\" --mode symbol # exact symbol\n" - " codelens search . \"handleChange\" --mode regex # regex code search\n" - " codelens search . \"MATCH (n) WHERE n.id CONTAINS x\" --mode graph\n" + " codelens search \"google auth\" . # semantic (default)\n" + " codelens search \"google auth\" . --mode symbol # exact symbol\n" + " codelens search \"handleChange\" . --mode regex # regex code search\n" + " codelens search \"MATCH (n) WHERE n.id CONTAINS x\" . --mode graph\n" + "\n" + "NOTE: pattern first, workspace second (opposite of other umbrellas).\n" + "Wrong order does not error — it silently searches for the workspace\n" + "path as the pattern and returns an empty result.\n" "\n" "For raw Cypher pass-through, prefer ``codelens graph ``." ) diff --git a/scripts/deadcode_engine.py b/scripts/deadcode_engine.py index 588c45ee..f111b0d2 100755 --- a/scripts/deadcode_engine.py +++ b/scripts/deadcode_engine.py @@ -2183,6 +2183,39 @@ def _detect_dead_from_registry( if not isinstance(nodes, list): return [] + # Rust unit tests live inline in the same file as production code inside + # `#[cfg(test)] mod tests { ... }`, annotated per-function with #[test] / + # #[tokio::test] / #[async_std::test] — unlike JS/Python where tests sit + # in a separate tests/ directory (already exempted below by + # _test_example_patterns). Without this, every Rust test function and + # every `mod tests` block false-positives as registry_dead, since the + # test harness invokes them via attribute discovery, not a CALLS edge. + _rust_test_attr_re = re.compile(r'^\s*#\[\s*(?:\w+::)?test\s*\]') + _rust_file_lines_cache: Dict[str, List[str]] = {} + + def _rust_symbol_is_test(file_path: str, line: int) -> bool: + if file_path not in _rust_file_lines_cache: + try: + abs_path = os.path.join(workspace, file_path) if not os.path.isabs(file_path) else file_path + with open(abs_path, 'r', encoding='utf-8', errors='ignore') as f: + _rust_file_lines_cache[file_path] = f.readlines() + except (IOError, OSError): + _rust_file_lines_cache[file_path] = [] + lines = _rust_file_lines_cache[file_path] + # Look at the 1-3 lines immediately above the definition for a + # #[test]-family attribute (allowing stacked attributes like + # #[should_panic] / #[ignore] between #[test] and the fn). + for offset in range(1, 4): + idx = line - 1 - offset + if idx < 0: + break + candidate = lines[idx] if idx < len(lines) else "" + if _rust_test_attr_re.match(candidate): + return True + if candidate.strip() and not candidate.strip().startswith('#['): + break + return False + for node in nodes: if not isinstance(node, dict): continue @@ -2236,6 +2269,14 @@ def _detect_dead_from_registry( # qualified names like `Module::func` if '::' in name and bare_name in _in_file_usages: continue + # Skip Rust inline test modules/functions (#[cfg(test)] mod tests, + # #[test] fn ...) — see _rust_symbol_is_test docstring above. + if file_path.endswith('.rs'): + if node_type == "module" and bare_name == "tests": + continue + if _rust_symbol_is_test(file_path, line): + continue + # Skip test fixtures and example files # v6.4: Expanded to catch examples/, e2e/, __tests__/, stories/ _test_example_patterns = [ diff --git a/scripts/query_graph_engine.py b/scripts/query_graph_engine.py index 858c8e18..f9620671 100644 --- a/scripts/query_graph_engine.py +++ b/scripts/query_graph_engine.py @@ -883,10 +883,8 @@ def execute_query( } # Add LIMIT - truncated = False if ast.limit is not None: sql += f" LIMIT {ast.limit}" - truncated = True # Execute try: @@ -925,6 +923,13 @@ def execute_query( raw_results = [_row_to_result(r, num_nodes) for r in rows] projected = _project_return(raw_results, ast.return_items, ast.return_star, ast.pattern.nodes) + # truncated means "results were actually cut off by LIMIT", not merely + # "the query happened to contain a LIMIT clause" — a LIMIT 5 query that + # only matches 1 row is not truncated (fixed: previously always True + # whenever ast.limit was set, misleading callers into thinking more + # results existed). + truncated = ast.limit is not None and len(rows) >= ast.limit + return { "status": "ok", "query": query, diff --git a/tests/test_codelens.py b/tests/test_codelens.py index 7a631811..eca3c749 100644 --- a/tests/test_codelens.py +++ b/tests/test_codelens.py @@ -406,6 +406,63 @@ def test_generic_lite_fallback(self): def test_non_dict_result_passthrough(self): self.assertEqual(_apply_lite("not a dict", "query"), "not a dict") + def test_summary_lite_trims_nested_top_items_and_flow_chain(self): + """Regression: summary's own job is anti-overload prioritized + findings, but --lite fell through to the generic fallback which + only trims the outer `findings` list, not each finding's nested + `top_items` (and dataflow findings nest a full flow_chain per + item) — a real workspace's --lite summary came back with + thousands of tokens of untouched detail.""" + result = { + "status": "ok", + "workspace": "/ws", + "findings": [ + { + "category": "dataflow_violations", + "total": 6, + "top_items": [ + {"source": {"file": "a.rs"}, "sink": {"file": "a.rs"}, + "flow_chain": [{"line": 1}, {"line": 2}]} + for _ in range(6) + ], + "action": "Add sanitizers", + }, + ], + } + lite = _apply_lite(result, "summary") + finding = lite["findings"][0] + self.assertEqual(len(finding["top_items"]), 3) + self.assertEqual(finding["top_items_total"], 6) + self.assertNotIn("flow_chain", finding["top_items"][0]) + + def test_history_lite_keeps_latest_and_trends(self): + """Regression: history's real payload (snapshots count, latest + snapshot's health metrics, trends, deltas) lives under keys the + generic fallback doesn't recognize, so --lite collapsed to just + {"status", "workspace"} with zero actual history data.""" + result = { + "status": "ok", + "workspace": "/ws", + "snapshots": 3, + "latest": { + "timestamp": "2026-07-12T00:00:00Z", + "health_score": 70, + "total_findings": 1291, + "findings_by_severity": {"critical": 116}, + "avg_complexity": 2.21, + "high_complexity_count": 21, + "irrelevant_internal_field": "should be dropped", + }, + "trends": {"health_score": [70, 70, 70]}, + "deltas": {"health_score": 0}, + } + lite = _apply_lite(result, "history") + self.assertEqual(lite["snapshots"], 3) + self.assertEqual(lite["latest"]["health_score"], 70) + self.assertNotIn("irrelevant_internal_field", lite["latest"]) + self.assertIn("trends", lite) + self.assertIn("deltas", lite) + def test_css_deep_lite(self): result = { "status": "ok", diff --git a/tests/test_command_registry.py b/tests/test_command_registry.py index 22979dae..8729d54d 100644 --- a/tests/test_command_registry.py +++ b/tests/test_command_registry.py @@ -41,10 +41,11 @@ def test_every_command_module_registers(): _DEPRECATED_ALIAS_MODULES = { "affected", "arch_metrics", "architecture", "binary_scan", "circular", "complexity", "dashboard", "dataflow", "dead_code", - "dependents", "diff", "env_check", "git_status", "graph_schema", - "import_snapshot", "init", "lsp_status", "orient", "outline", - "ownership", "perf_hint", "query_graph", "regex_audit", "secrets", - "side_effect", "smell", "staleness", "taint", "trace", "vuln_scan", + "dependents", "diff", "env_check", "export_snapshot", "git_status", + "graph_schema", "import_snapshot", "init", "lsp_status", "orient", + "outline", "ownership", "perf_hint", "query_graph", "regex_audit", + "secrets", "side_effect", "smell", "staleness", "taint", "trace", + "vuln_scan", } _UTILITY_MODULES |= _DEPRECATED_ALIAS_MODULES missing = [] diff --git a/tests/test_deadcode_engine.py b/tests/test_deadcode_engine.py index 24ba26e7..f33a9979 100644 --- a/tests/test_deadcode_engine.py +++ b/tests/test_deadcode_engine.py @@ -623,3 +623,78 @@ def test_registry_dead_without_same_file_usages_flags_everything(self): ) finally: shutil.rmtree(ws, ignore_errors=True) + + def test_registry_dead_exempts_rust_inline_test_functions(self): + """Rust unit tests live inline as `#[cfg(test)] mod tests { #[test] + fn ... }` in the same file as production code — unlike JS/Python + where tests sit in a separate tests/ directory. A #[test]-attributed + function has ref_count==0 by design (invoked by the test harness via + attribute discovery, not a CALLS edge) and must not be flagged, nor + should the `mod tests` module itself. A genuinely dead production + function in the same file must still be flagged. + """ + import json + ws = tempfile.mkdtemp() + try: + rs_content = ( + "fn genuinely_unused_prod_fn() {}\n" + "\n" + "#[cfg(test)]\n" + "mod tests {\n" + " use super::*;\n" + "\n" + " #[test]\n" + " fn atomic_write_creates_file() {\n" + " assert!(true);\n" + " }\n" + "\n" + " #[tokio::test]\n" + " async fn async_test_case() {\n" + " assert!(true);\n" + " }\n" + "}\n" + ) + with open(os.path.join(ws, "verify.rs"), 'w') as f: + f.write(rs_content) + + registry = { + "nodes": [ + { + "fn": "genuinely_unused_prod_fn", "file": "verify.rs", "line": 1, + "ref_count": 0, "status": "dead", "type": "function", "pub": False + }, + { + "fn": "tests", "file": "verify.rs", "line": 4, + "ref_count": 0, "status": "dead", "type": "module", "pub": False + }, + { + "fn": "atomic_write_creates_file", "file": "verify.rs", "line": 8, + "ref_count": 0, "status": "dead", "type": "function", "pub": False + }, + { + "fn": "async_test_case", "file": "verify.rs", "line": 13, + "ref_count": 0, "status": "dead", "type": "function", "pub": False + }, + ], + "edges": [] + } + codelens_dir = os.path.join(ws, ".codelens") + os.makedirs(codelens_dir, exist_ok=True) + with open(os.path.join(codelens_dir, "backend.json"), 'w') as f: + json.dump(registry, f) + + result = _detect_dead_from_registry(ws, None) + names = {item["name"] for item in result} + + assert "tests" not in names, f"mod tests must be exempted. Findings: {result}" + assert "atomic_write_creates_file" not in names, ( + f"#[test] fn must be exempted. Findings: {result}" + ) + assert "async_test_case" not in names, ( + f"#[tokio::test] fn must be exempted. Findings: {result}" + ) + assert "genuinely_unused_prod_fn" in names, ( + f"Genuinely dead production fn must still be flagged. Findings: {result}" + ) + finally: + shutil.rmtree(ws, ignore_errors=True) diff --git a/tests/test_export_snapshot.py b/tests/test_export_snapshot.py new file mode 100644 index 00000000..54255732 --- /dev/null +++ b/tests/test_export_snapshot.py @@ -0,0 +1,150 @@ +"""Tests for the export-snapshot command (issue #218). + +Companion to import-snapshot: import-snapshot was permanently broken since +issue #195 dropped the standalone export-snapshot command without leaving +any way to produce the .codelens.gz file it reads. These tests verify the +new `deps --check export-snapshot` sub-mode writes a snapshot that +`deps --check import-snapshot` can load back (round trip), and that the +bare `codelens deps ` default does not attempt either +snapshot check (they are side-effecting, opt-in only). +""" + +import os +import shutil +import sqlite3 +import sys +import tempfile + +import pytest + +SCRIPT_DIR = os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "scripts" +) +if SCRIPT_DIR not in sys.path: + sys.path.insert(0, SCRIPT_DIR) + +from commands.export_snapshot import cmd_export_snapshot # noqa: E402 +from commands.import_snapshot import cmd_import_snapshot # noqa: E402 +from commands.deps import _parse_checks, ALL_CHECKS # noqa: E402 + + +@pytest.fixture +def scanned_workspace(): + """A workspace with a minimal graph DB already populated.""" + tmpdir = tempfile.mkdtemp(prefix="codelens_export_snapshot_test_") + codelens_dir = os.path.join(tmpdir, ".codelens") + os.makedirs(codelens_dir, exist_ok=True) + db_path = os.path.join(codelens_dir, "codelens.db") + + conn = sqlite3.connect(db_path) + conn.executescript(""" + CREATE TABLE graph_nodes ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + node_id TEXT NOT NULL UNIQUE, + node_type TEXT NOT NULL DEFAULT 'function', + name TEXT NOT NULL, + file TEXT, + line INTEGER, + extra_json TEXT + ); + CREATE TABLE graph_edges ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source_id TEXT NOT NULL, + target_id TEXT, + edge_type TEXT NOT NULL, + file TEXT, + line INTEGER, + confidence REAL NOT NULL DEFAULT 1.0, + extra_json TEXT + ); + """) + conn.execute( + "INSERT INTO graph_nodes (node_id, node_type, name, file, line) " + "VALUES ('a.py:1:foo', 'function', 'foo', 'a.py', 1)" + ) + conn.execute( + "INSERT INTO graph_nodes (node_id, node_type, name, file, line) " + "VALUES ('a.py:5:bar', 'function', 'bar', 'a.py', 5)" + ) + conn.execute( + "INSERT INTO graph_edges (source_id, target_id, edge_type, file, line) " + "VALUES ('a.py:1:foo', 'a.py:5:bar', 'CALLS', 'a.py', 2)" + ) + conn.commit() + conn.close() + + yield tmpdir + shutil.rmtree(tmpdir, ignore_errors=True) + + +class TestExportSnapshot: + def test_export_creates_gz_file(self, scanned_workspace): + result = cmd_export_snapshot(scanned_workspace) + assert result["status"] == "ok" + assert os.path.isfile(result["output_path"]) + assert result["bytes_written"] > 0 + assert result["header"]["node_count"] == 2 + assert result["header"]["edge_count"] == 1 + + def test_export_missing_db_returns_error(self): + empty_ws = tempfile.mkdtemp(prefix="codelens_export_snapshot_empty_") + try: + result = cmd_export_snapshot(empty_ws) + assert result["status"] == "error" + assert "error" in result + finally: + shutil.rmtree(empty_ws, ignore_errors=True) + + def test_export_custom_output_path(self, scanned_workspace): + custom_path = os.path.join(scanned_workspace, "custom.codelens.gz") + result = cmd_export_snapshot(scanned_workspace, output_path=custom_path) + assert result["status"] == "ok" + assert result["output_path"] == custom_path + assert os.path.isfile(custom_path) + + def test_export_then_import_round_trip(self, scanned_workspace): + """The core issue #218 regression: export must produce a file that + import-snapshot can actually load, restoring the same node/edge + counts into a fresh database.""" + export_result = cmd_export_snapshot(scanned_workspace) + assert export_result["status"] == "ok" + + # Import into a brand-new empty workspace/db (simulates a teammate + # loading the shared snapshot without ever running `scan`). + fresh_ws = tempfile.mkdtemp(prefix="codelens_export_snapshot_fresh_") + try: + os.makedirs(os.path.join(fresh_ws, ".codelens"), exist_ok=True) + shutil.copy( + export_result["output_path"], + os.path.join(fresh_ws, ".codelens", "snapshot.codelens.gz"), + ) + + import_result = cmd_import_snapshot(fresh_ws) + assert import_result["status"] == "ok" + assert import_result["total_inserted"] == 3 # 2 nodes + 1 edge + assert import_result["header"]["node_count"] == 2 + assert import_result["header"]["edge_count"] == 1 + finally: + shutil.rmtree(fresh_ws, ignore_errors=True) + + +class TestDepsDefaultExcludesSnapshotChecks: + """import-snapshot/export-snapshot must not run in the bare + `codelens deps ` default — they are side-effecting and + always fail with no explicit --input/--output, which would make every + default `deps` run show a spurious error entry.""" + + def test_default_check_list_excludes_snapshot_checks(self): + checks = _parse_checks(None) + assert "import-snapshot" not in checks + assert "export-snapshot" not in checks + assert "affected" in checks + assert "dependents" in checks + assert "circular" in checks + + def test_explicit_check_still_allows_snapshot_checks(self): + assert _parse_checks("export-snapshot") == ["export-snapshot"] + assert _parse_checks("import-snapshot") == ["import-snapshot"] + + def test_export_snapshot_registered_in_all_checks(self): + assert "export-snapshot" in ALL_CHECKS diff --git a/tests/test_query_graph.py b/tests/test_query_graph.py index a24cd010..9a0d522d 100644 --- a/tests/test_query_graph.py +++ b/tests/test_query_graph.py @@ -573,6 +573,18 @@ def test_truncated_flag(self, graph_db): r2 = execute_query("MATCH (f:Function) RETURN f.name", ws, db_path=db_path) assert r2["truncated"] is False + def test_truncated_flag_false_when_limit_exceeds_match_count(self, graph_db): + """LIMIT present but under-filled must not report truncated=True + (regression: truncated used to be set whenever ast.limit was not + None, regardless of whether any rows were actually cut off).""" + ws, db_path = graph_db + r = execute_query( + "MATCH (f:Function) WHERE f.name = 'main' RETURN f.name LIMIT 50", + ws, db_path=db_path, + ) + assert r["count"] < 50 + assert r["truncated"] is False + # ─── CLI command registration ──────────────────────────────────────────────