Feature request: --max-memory flag for RAM-constrained hosts

[darkhipo]

On an 8GB M1 Air indexing the MultiAgency codebase (~6000 files), codebase-memory-mcp v0.8.1 consumes 8-11GB RSS. Two concurrent instances pushed the host to swap death with malloc corruption and SIGSEGV.

The knowledge graph is built fully in memory. For large codebases on constrained hardware, this causes:

• Swap thrash (load averages > 20)
• malloc(3) failures ("pointer being freed was not allocated")
• Process crashes from corrupted allocations

Request: a --max-memory=N flag (in MB or GB) that limits the in-memory graph size. When exceeded, spill to disk or use a streaming/paged approach. This would let the binary run on 8GB machines by trading speed for RAM safety.

Workaround: kill and restart periodically. Not sustainable.

[RithvikReddy0-0]

Took a look at how memory is currently bounded, in case it helps scope this.

There's already an RSS budget mechanism (cbm_mem_init in foundation/mem.c, currently MAIN_RAM_FRACTION = 50% of total RAM) that drives extraction back-pressure in pass_parallel.c it sheds per-file retention when the process goes over budget. So a --max-memory=N flag could feed straight into that path with little plumbing.

The caveat is that the existing budget bounds the extraction peak, not the final knowledge graph, which is held fully in memory before the dump to SQLite. On a ~6000-file repo that final graph is most of your 8-11GB. So a flag wired only into cbm_mem_init would tighten back-pressure but wouldn't actually keep the process under N on a large repo the graph would still blow past it. Delivering the real "run safely on 8GB" behavior needs the paged/spill approach you described, which is the larger change.

Separately, the malloc corruption "pointer being freed was not allocated" plus SIGSEGV — is worth treating as its own issue. That's heap corruption, not a clean allocation failure; OOM alone usually gives a NULL malloc or the OOM-killer, not a freed-pointer abort. It may be a latent double-free/overrun that only trips under pressure (or under the two-concurrent-instances case, if any state is shared via the cache dir). Might be worth splitting that out so it doesn't get lost behind the feature work.

Happy to help with the flag-into-cbm_mem_init piece if you decide a partial cap (with honest docs about what it does and doesn't bound) is worth shipping ahead of the full paged-graph work.

[DeusData]

Thanks. I’ve linked this under #593. An explicit memory budget should line up with worker-count sizing and cgroup/container detection, and it needs a clear failure/degradation mode when the requested budget is exceeded.

[Tutul-]

Btw, it's not only for "memory budget", very large project (like the Unreal Engine source code) easily blow up the indexing memory usage.
For that kind of large repository (~36G compared to the ~7.5G of the Linux kernel source code) it consumed my full 32G of RAM.

[DeusData]

Thanks, that is useful confirmation that this is not only an 8GB-machine concern. I am raising the priority while keeping it in 0.9.2-rc because the durable fix is likely architectural: bounded graph memory, spill/paging behavior, and clear degradation rather than only a thin flag. If you can share version, command line, and whether the run completed or failed on the Unreal-sized checkout, that would help size the benchmark target.