pkgmap/envscan repo walks bypass gitignore exclusions - 15+ min apparent hang on repos with large ignored artifact dirs

[Sowiedu]

Summary

index_repository takes 15–18 minutes (appearing completely hung) on a repo that actually indexes in ~3 seconds, because the auxiliary filesystem walks in pkgmap and envscan do not honor the project's gitignore-derived exclusions. They descend into a 1.6-million-file gitignored artifact directory that the discovery pass correctly excludes.

Environment

• codebase-memory-mcp built from main @ dcf98dc (also reproduced on release v0.8.1)
• Windows 11 Pro (26200), NTFS, Ryzen 7 8700F
• Repo: ~1,000 source files (TS/React + Python), plus a gitignored map-import/output/ containing 1,638,518 generated map-tile files

Symptoms

• parallel.extract.progress reaches done=1020 total=1021 and then no log output for 15+ minutes while the process burns ~100% of one core, RAM slowly grows, disk I/O ≈ 0. Looks exactly like a deadlock/hang. (In reality extraction has finished — 1021 isn't a multiple of 10, so the final progress tick never prints.)
• The wall time is finite but enormous; the process eventually completes if you wait long enough.

Root cause

At the extraction join, merge_pkg_entries() in src/pipeline/pass_parallel.c runs:

/* Supplement with a repo-wide filesystem walk so manifests filtered
 * by the main discoverer (package.json, composer.json — in
 * IGNORED_JSON_FILES) still feed pkgmap. */
cbm_pkgmap_scan_repo(ctx->repo_path, &pkg_entries[0]);

pkgmap_walk_dir() (src/pipeline/pass_pkgmap.c) only skips cbm_should_skip_dir() — the hardcoded name list (node_modules, .git, …). It never consults the project's gitignore-derived exclusion list, which discovery itself honors (index_status correctly reports map-import/output in excluded.dirs). A directory named output matches nothing in the hardcoded list, so the walk enumerates all 1.6M files, single-threaded, invisibly between two log lines.

pass_envscan.c has the same defect with its own private is_ignored_dir() hardcoded list.

Diagnosis evidence

Sysinternals handle64 -p <pid> on the "hung" process, captured mid-stall:

  7D0: File          C:\...\tacops\map-import\output\camp_pendleton\0\248
  A94: File          C:\...\tacops\map-import\output
  B18: File          C:\...\tacops\map-import\output\camp_pendleton\0
  CAC: File          C:\...\tacops\map-import\output\camp_pendleton

A/B confirmation: a git clone of the same repo (which lacks the untracked tile tree) full-indexes in 3.0 s; the original working copy takes 15–18 min. With the exclusion fix applied to both walks (PR to follow), the original working copy full-indexes in 3.0 s and an incremental no-op takes 0.4 s.

Repro

1. Any repo with a large gitignored directory (ML checkpoints, tile pyramids, media assets):

   mkdir big && git init
   echo "bigdir/" > .gitignore
   # generate a few hundred thousand small files under bigdir/
   

2. index_repository it. Discovery excludes bigdir/ (visible in index_status), but wall time scales with the total file count of bigdir/ anyway; on NTFS this is minutes for ~10^6 files.

Possibly related

The symptom profile (extraction "stalls" near the end, RAM growth, no I/O, hours of apparent hang on repos with heavy artifact directories) may explain parts of #775 and the large-repo reports under #593.

Suggested fix

Thread the pipeline ctx's excluded-dirs list into cbm_pkgmap_scan_repo() and the envscan walker and prefix-match relative paths before descending — PR incoming from this account.

[DeusData]

Thanks, this is a strong performance repro. Triage: high-priority large-repo/indexing performance under #593.

The key acceptance criterion is that every auxiliary repo walk, including pkgmap/envscan, must share the same exclusion model as discovery. A gitignored artifact tree should not be walked after discovery has excluded it. I see #793 is already opened against this; we will review it against this issue and make sure the fixture covers the bypass rather than only the runtime symptom.