fast mode silently excludes `scripts/` — first-class Python source dropped from graph (built-in skip list?)

[Limebird01]

Version

codebase-memory-mcp 0.9.0

Platform

Linux (x64)

Install channel

GitHub release archive / install.sh / install.ps1

Binary variant

standard (portable static build, linux-amd64)

What happened, and what did you expect?

index_repository --mode fast silently excluded the entire scripts/ directory of a mid-size Python repo. Only 46 of 156 .py files were indexed, and scripts appeared in the excluded.dirs list of the index output. Re-indexing the same repo with --mode full captured everything (5,286 nodes / 17,328 edges, 6.0 s, ~232 MB max RSS).

No .gitignore or .cbmignore pattern matches scripts/. In this repo (as in many Python projects) scripts/ holds first-class production entry-point modules — it is a common Python layout, not a bin of throwaway files.

Expected: fast mode should reduce analysis depth (cheaper resolution strategies, fewer passes), not silently drop first-class source directories from the graph — or at minimum it should emit a visible warning that a top-level source dir was skipped by a built-in heuristic.

This looks like the same silent-exclusion family as #1184 (built-in skip list drops deploy/deployment(s) dirs, mode full, not overridable via .cbmignore) and #411 (closed; moderate mode silently drops subtrees), and #1219 (default assets exclusion swallows API route files) — but none of those covers scripts/ in fast mode specifically. Note the docs state the built-in skip list is not user-overridable, so there is currently no workaround other than switching to --mode full.

Reproduction

1. Code: any generic Python repo with a top-level scripts/ directory containing real importable modules, e.g.:

   repo/
     src/pkg/core.py
     scripts/deploy_tool/main.py   # first-class entry points, plain .py modules
     scripts/reporting/report.py
   

2. Commands:

   codebase-memory-mcp cli index_repository --repo-path /path/to/repo --mode fast --persistence false
   codebase-memory-mcp cli index_repository --repo-path /path/to/repo --mode full --persistence false
   

3. Result: fast-mode output lists scripts under excluded.dirs; indexed file count drops to 46/156 .py; every symbol under scripts/ is absent from the graph. Full mode indexes all 156 files. Expected: identical file coverage in both modes (only analysis depth should differ), or an explicit warning naming the skipped directory.

Logs

Diagnostics trajectory (memory / performance / leak issues)

Project scale (if relevant)

5,286 nodes / 17,328 edges / 156 .py files (full mode)

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.

[AmirF194]

Reproduced on current main (d6be58ef, 2026-07-31), clean container. Minimal repo: src/pkg/core.py plus scripts/deploy_tool/main.py and scripts/reporting/report.py (each one importable function). fast mode:

{"project":"r1406fast","excluded":{"dirs":["scripts"],"count":1,...},"nodes":19,"edges":17,...}

full mode on the same tree: nodes":28,"edges":49 with nothing excluded, i.e. the two scripts/ functions and their edges only show up in full.

Root cause: src/discover/discover.c, FAST_SKIP_DIRS[] (around line 59) hardcodes "scripts" alongside "docs", "examples", "samples", "fixtures", etc, and cbm_should_skip_dir() applies that list whenever mode != CBM_MODE_FULL. There's no separate signal for "looks like CI/deploy tooling" vs "looks like a source package"; scripts/ is skipped by name alone. History-wise, the whole list (including this entry) came in as one bulk add in 94a9a92f ("Add C-native pipeline..."), with nothing that specifically calls out scripts/ versus its neighbors, so this reads as an oversight in the original list rather than an intentional exclusion.

Competing-PR sweep on src/discover/discover.c (87/87 open PRs, uncapped): 6 candidates touch the file. Two touch FAST_SKIP_DIRS itself: #1166 only adds an unrelated ALWAYS_SKIP_DIRS entry (.codebase-memory) and #1245 (a large incremental-indexing rewrite) reorders the same list but keeps "scripts" in it verbatim. Neither removes or gates it. #1203 is unrelated (shebang-based extensionless-script detection).

[DeusData]

Thank you for identifying a concrete false-negative caused by fast mode. Current discovery code includes scripts and tools in the fast-mode directory skip list, so first-class source under those conventional directories is silently excluded. I have removed the unrelated Windows label and routed this as a high-priority parsing/UX bug in 0.9.1-rc.

[mauro33]

Corroborating this from a Go repo, with three generalizations. Your title asks "built-in skip list?" and I think this answers it.

Version: 0.9.0 on Windows. Index built by our own tooling at mode moderate, not fast. So whatever the skip list is, it is not a fast-mode optimization.

Two first-party trees are absent from the graph in a Go repo:

internal/core/external/    26 .go files
docs/                      72 .md files

Evidence that the index is otherwise healthy, and that these two trees specifically are skipped:

search_graph file_pattern='internal/core/doctor'   -> 158 nodes      (sibling package, fine)
search_graph qn_pattern='.*\.external\..*'         -> total 0
search_graph file_pattern='^docs/'                 -> total 0

And the same absence in search_code, so it is not only the graph:

git grep -c Provisioner                               -> 82 hits across 25 files,
                                                         16 of them in internal/core/external/external.go
search_code pattern='Provisioner' mode='files'        -> total_grep_matches 19, ZERO from that tree
search_code pattern='Provisioner' file_pattern='*.go' -> still zero from that tree

One trap worth naming, because it is the obvious way to dismiss the docs/ half: this is NOT "markdown is not indexed". Markdown from other trees IS read.

search_code pattern='growth-axes' mode='files'
  -> returns internal/packs/skills/sharpen/SKILL.md,
             internal/packs/skills/tool-research/SKILL.md,
             specs/002-graphify-full-absorb/plan.md
git grep growth-axes
  -> additionally matches docs/how-it-works.md, docs/internal/growth-axes.md,
     docs/internal/cerebro-golden-rules.md and 7 more

So .md is fine; docs/ is not.

A detail that made this hard to notice, and that may help others confirm it: search_graph file_pattern='internal/core/external' returns results, because it prefix-matches a DIFFERENT package named internal/core/externalrules/. A casual check looks like the tree is present.

Why this class of skip hurts more than it helps: external and docs are plausible entries for skipping vendored or generated content, but they are also ordinary first-party names. In this repo internal/core/external/ is where every external-tool adapter lives, so both search_code and search_graph answer "no such code" for an entire architectural layer, silently and confidently.

Two asks, in priority order:

1. Report what was skipped. index_status completing with no mention of dropped trees is what makes this a silent failure rather than a visible limitation. Even a count would have saved the investigation.
2. Make the list configurable, or match on path shape (vendor/, node_modules/, generated markers) rather than on a bare directory name at any depth.

Happy to run any probe you want against this repo if it helps narrow it.