Full-repo indexing crashes in parallel.extract on large AOSP-scale repository

[sunny5541]

codebase-memory-mcp crashes during index_repository on a very large AOSP-scale repository.

The failure happens in parallel.extract with:

Segmentation fault (core dumped)

This still reproduces after:

- excluding heavy generated/build directories with .cbmignore
- switching to mode: "fast"
- reducing the discovered file set significantly

## Version / source info

This is a locally built dev binary from source, not a tagged release binary.

- binary version: codebase-memory-mcp dev
- source checkout commit: e6e9c58e80a808d9a8450b11dc6c8dde085c8417
- source checkout branch: main


## Environment

- OS: Ubuntu 22.04.5 LTS
- kernel: Linux ... 6.8.0-106-generic ... x86_64
- glibc: 2.35
- gcc: 11.4.0
- g++: 11.4.0
- make: 4.3
- clang not installed in this environment
- cmake not installed in this environment

## Reproduction

codebase-memory-mcp cli index_repository '{"repo_path":"/absolute/path/to/large/aosp-style/repo"}'

Also reproduces with:

codebase-memory-mcp cli index_repository '{"repo_path":"/absolute/path/to/large/aosp-style/repo","mode":"fast"}'

Example exclusions used during investigation:

out/
out/**
prebuilts/
prebuilts/**
kernel/prebuilts/
kernel/prebuilts/**
external/XNNPACK/test/
external/XNNPACK/test/**
external/libabigail/tests/
external/libabigail/tests/**
packages/modules/NeuralNetworks/runtime/test/
packages/modules/NeuralNetworks/runtime/test/**
packages/modules/NeuralNetworks/tools/systrace_parser/parser/test/
packages/modules/NeuralNetworks/tools/systrace_parser/parser/test/**
packages/apps/TV/tuner/tests/
packages/apps/TV/tuner/tests/**

## Observed scale

- before exclusions: about 847196 files
- after excluding out/: about 796478 files
- after more aggressive exclusions + mode:"fast": about 228794 files

Even at the smaller count, the process still crashes in parallel.extract.

## Representative log

level=info msg=parallel.extract.file.done pos=547 elapsed_ms=5895 defs=141 path=art/compiler/optimizing/load_store_elimination_test.cc
level=info msg=parallel.extract.progress done=820 total=228794
level=info msg=parallel.extract.progress done=830 total=228794
level=info msg=parallel.extract.file.done pos=495 elapsed_ms=7780 defs=196 path=kernel-6.1/drivers/net/wireless/ralink/rt2x00/rt2800lib.c
level=info msg=parallel.extract.progress done=840 total=228794
level=info msg=parallel.extract.progress done=850 total=228794
level=info msg=parallel.extract.progress done=860 total=228794
level=info msg=parallel.extract.progress done=870 total=228794
level=info msg=parallel.extract.file.done pos=621 elapsed_ms=5011 defs=387 path=art/compiler/optimizing/code_generator_arm64.cc
level=info msg=parallel.extract.progress done=880 total=228794
level=info msg=parallel.extract.progress done=890 total=228794
level=info msg=parallel.extract.progress done=900 total=228794
level=info msg=parallel.extract.progress done=910 total=228794
level=info msg=parallel.extract.progress done=920 total=228794
level=info msg=parallel.extract.progress done=930 total=228794
level=info msg=parallel.extract.progress done=940 total=228794
level=info msg=parallel.extract.progress done=950 total=228794
level=info msg=parallel.extract.progress done=960 total=228794
level=info msg=parallel.extract.progress done=990 total=228794
level=info msg=parallel.extract.progress done=970 total=228794
level=info msg=parallel.extract.progress done=1000 total=228794
level=info msg=parallel.extract.progress done=1010 total=228794
Segmentation fault (core dumped)

## Additional observations

- The last files printed before the crash vary between runs.
- A miniature repo built from a few “last logged” files indexed successfully.
- Shard indexing works reliably when the large repo is split into smaller subtrees.

This makes it look more like a scalability / concurrency / cumulative-memory problem than a single bad file.

## Initial source-level observations

These seem relevant from the current source tree:

- initial indexing uses all cores: cbm_default_worker_count(true)
- large repos almost always use parallel extraction (MIN_FILES_FOR_PARALLEL = 50)
- memory budget appears to be logged, not enforced
- worker threads use fixed 8 MB stacks
- result_cache retains extracted per-file results across extraction / registry / resolve
- CBM_EXTRACT_BUDGET is a parse timeout, not a memory bound
- fast mode still uses the same parallel extraction path

## Request

Is this a known scalability limitation in the current dev build?

The most promising fixes seem to be:

1. add a user-facing max_workers option to index_repository
2. enforce memory-based backpressure instead of only logging budget
3. batch or stream result_cache instead of retaining results for the whole repo
4. provide a safer fallback for huge repositories

[DeusData]

Thanks @sunny5541 for reporting. The segfault in parallel.extract on AOSP-scale repos is likely a memory pressure issue in the worker pool. Could you share the repo size (file count, LOC) and your system specs (RAM, OS)?

[sunny5541]

Thanks @sunny5541 for reporting. The segfault in parallel.extract on AOSP-scale repos is likely a memory pressure issue in the worker pool. Could you share the repo size (file count, LOC) and your system specs (RAM, OS)?

Sure. Here are the numbers from the environment where the crash reproduces.

System:

• OS: Ubuntu 22.04.5 LTS
• Kernel: Linux 6.8.0-106-generic x86_64
• RAM: 31 GiB
• Swap: 63 GiB
• The tool selected 20 workers for initial indexing on this machine

Repository scale:

• On-disk size: about 363 GB
• Total file count under repo root: 2,023,342 files

Indexing-specific file counts I observed:

• before exclusions: ~847,196 discovered files
• after excluding out/: ~796,478 discovered files
• after more aggressive exclusions by .cbmignore + mode="fast": ~228,794 discovered files

[mrjared16]

Thanks @sunny5541 for reporting. The segfault in parallel.extract on AOSP-scale repos is likely a memory pressure issue in the worker pool. Could you share the repo size (file count, LOC) and your system specs (RAM, OS)?

it crashed with this repo https://github.com/apache/incubator-seata/ on my linux 16gb ram machine

[kuba6000]

I got it to crash with a singular file in the entire repo

Here's the file for testing
https://github.com/GTNewHorizons/GT5-Unofficial/blob/7a31a1dae6e3ef36b36cc38e7b3438637b1cadd8/src/main/java/gregtech/common/tileentities/machines/multi/MTENanoForge.java

[DeusData]

The parallel.extract crash class has had substantial hardening since this report: growable AST stacks (no silent fixed-size overflow → heap corruption) (#339), and multiple extraction/LSP OOB+NULL-deref fixes (#215/#312/#360). Large real-world C++ repos (KiCad ~6.3k files, Icinga2) now index cleanly under AddressSanitizer in our CI.

Could you confirm whether the crash still reproduces on v0.7.0 / current main? If yes, please share which repo (or a representative subtree) + the last parallel.extract.file.start log line before the crash — that names the file it dies on and lets me add a precise regression test. 🙏

[DeusData]

Thanks — a segfault in parallel.extract on an AOSP-scale repository is a memory/scale failure in the parallel extraction path, an area substantially hardened for v0.7.0: #339 (converted remaining fixed-size stacks in extract to dynamic allocation), the lsp_cross scale fixes #340/#344, and 82d5c84 (thread-parser/slab ordering in the cross loop).

AOSP is impractical for me to clone in full, so I can't reproduce your exact repo here. Could you retest on v0.7.0? If it still segfaults in parallel.extract, the most actionable thing would be: which file position/path it crashes on (the parallel.extract.file.start pos=… path=… line just before the crash), and whether a single large generated file is implicated — that lets me build a targeted test without the whole tree. Keeping open pending your v0.7.0 result. Thanks!

[Pxx500]

0.7.0 seems to be working fine on the old repo reported by kuba, so that's fixed

[DeusData]

Confirmed working on v0.7.0 (thanks @Pxx500 for verifying on kuba's repo) — closing as fixed. 🙏

The v0.7.0 hardening of the parallel extraction path addressed this: #339 (remaining fixed-size stacks in extract converted to dynamic allocation), the lsp_cross scale fixes #340/#344, and 82d5c84 (thread-parser/slab ordering in the cross loop).

@sunny5541 — if a full AOSP index still segfaults in parallel.extract on v0.7.0, please reopen with the parallel.extract.file.start pos=… path=… line just before the crash (which file it dies on) and I'll build a targeted repro from that single file. Thanks both!