Submodule visibility gap and poll-interval scaling in git-strategy watcher

[halindrome]

Summary

The git-strategy watcher in internal/watcher/watcher.go has two related problems that compound for large projects with git submodules.

Problem 1: Submodule working-tree changes are invisible until committed

The gitSentinel function runs git status --porcelain on the project root:

cmd := exec.CommandContext(ctx, "git", "--no-optional-locks", "-C", rootPath,
    "status", "--porcelain", "--untracked-files=normal")

For a repo with submodules, this shows submodule directories as modified only after a commit is made inside the submodule (causing the submodule HEAD to diverge from what the parent records). Uncommitted edits to files inside a submodule are invisible to the parent's git status — the parent only sees the submodule as a single opaque directory entry.

This means:

• Agent edits files in apps/mobile/ → parent git status shows nothing → no reindex triggered
• Agent commits in apps/mobile/ → parent git status shows apps/mobile as modified → reindex eventually triggered (up to 60s later)
• Unstaged changes inside submodules are never visible to the watcher

Problem 2: Poll interval backs off to 60s for large repos

The adaptive interval formula:

func pollInterval(fileCount int) time.Duration {
    ms := 5000 + (fileCount/500)*1000
    if ms > 60000 { ms = 60000 }
    return time.Duration(ms) * time.Millisecond
}

A monorepo with 6 submodules can easily have 25,000+ files, which pushes the interval to the 60-second cap. Combined with Problem 1, this means:

• Uncommitted changes in submodules: never detected
• Committed changes in submodules: up to 60s delay before reindex

Suggested improvements

Option A: Add --recurse-submodules to gitSentinel

cmd := exec.CommandContext(ctx, "git", "--no-optional-locks", "-C", rootPath,
    "status", "--porcelain", "--untracked-files=normal", "--recurse-submodules")

This makes working-tree changes inside submodules visible to the parent. The downside is that git status --recurse-submodules is significantly slower for large repos.

Option B: Run per-submodule git sentinels

Parse .gitmodules and run a separate gitSentinel for each submodule path. Allows the same efficient git status per submodule without recursion overhead. More complex to implement.

Option C: Signal / touch mechanism

Provide a way for external processes (hooks, scripts) to signal the daemon that a reindex is needed immediately — e.g., a sentinel file path or a lightweight HTTP endpoint. This would allow Claude Code hooks to trigger reindexing on demand without waiting for the next poll cycle.

Context

This was discovered while building a Claude Code hook layer on top of codebase-memory-mcp for a monorepo with 6 submodules (mobile app, web app, REST API, SSE service, chatbot orchestrator, MCP server). The 60-second lag and submodule blindspot make the auto-sync feature unreliable for this use case — we work around it by calling index_repository explicitly after commits, but the gap is still visible.

Environment

• codebase-memory-mcp: latest (reviewed source at commit on main)
• Project type: git repo with 6 submodules, ~25k+ files
• Watch strategy selected: git (git repo detected correctly)

[halindrome]

Deep-dive findings on the watcher implementation

After tracing through the actual watcher code in src/watcher/watcher.c, I found some important details that significantly simplify the fix for this issue. Sharing here for discussion.

Key finding: git_is_dirty() is a boolean check, not a parser

The git_is_dirty() function (line 119) does not parse the output of git status --porcelain. It reads exactly one line via fgets() and checks if it's non-empty:

char line[256];
bool dirty = false;
if (fgets(line, sizeof(line), fp)) {
    /* Any output means changes */
    size_t len = strlen(line);
    while (len > 0 && (line[len - 1] == '\n' || line[len - 1] == '\r')) {
        line[--len] = '\0';
    }
    if (len > 0) {
        dirty = true;
    }
}

It returns a boolean — dirty or not. It doesn't extract filenames, status codes, or distinguish submodule entries from regular file entries.

This means adding --recurse-submodules to the git status command requires zero parsing changes. Any non-empty output (whether from the parent or a submodule) triggers the dirty flag, which triggers a full reindex.

Key finding: the indexer already walks submodule content

The discover module's walk_dir() in src/discover/discover.c recurses into all directories that aren't in the hardcoded skip list (.git, node_modules, .venv, etc.). Submodule directories appear as regular directories on the filesystem and are not in the skip list, so they are already indexed during every reindex.

The problem is entirely in the watcher's change detection gate — not in the indexer itself.

Key finding: full reindex is already the behavior for any change

The watcher triggers index_fn() which runs a complete filesystem traversal via walk_dir(). There is no incremental or scoped reindex — any dirty flag causes a full reindex of the entire project. This means the concern that "a single change deep in a submodule triggers reindex of entire parent repo" is already the existing behavior for any change, not new overhead from --recurse-submodules.

Revised assessment of the three options

Option A (--recurse-submodules) — now strongly recommended:

Given the findings above, this is a one-line change:

snprintf(cmd, sizeof(cmd),
         "git --no-optional-locks -C '%s' status --porcelain "
         "--untracked-files=normal --recurse-submodules 2>/dev/null",
         root_path);

• No parsing changes needed (boolean check)
• No new reindex overhead (full reindex is already the behavior)
• Automatic detection of all submodule changes (uncommitted, staged, untracked)
• Works for nested submodules
• Backwards compatible (flag ignored if no submodules)

The only cost is a slightly slower git status call on each poll, which is negligible on a 60-second interval.

Option B (per-submodule sentinels) — no longer recommended for this issue:

This adds ~1000 LOC of complexity (.gitmodules parser, per-submodule watchers, partial indexing) to avoid a full reindex that already happens on every change. The complexity isn't justified when Option A achieves the same result with one line. Option B could become relevant if per-submodule incremental indexing is added later, but that's a separate architectural change.

Option C (expose cbm_watcher_touch()) — recommended as complement to A:

cbm_watcher_touch() already exists but isn't exposed to MCP clients. It resets next_poll_ns = 0, causing the next 5-second watcher tick to run check_changes() immediately instead of waiting for the adaptive interval (up to 60s).

Important nuance: touch alone is insufficient without Option A. Touch triggers an immediate check_changes(), but check_changes() calls git_is_dirty() — which without --recurse-submodules is still blind to submodule working-tree changes. With Option A in place, touch becomes a way to bypass the poll interval for on-demand reindex from hooks.

The adaptive interval provides natural debounce:

1. index_fn() is synchronous (blocks the poll loop)
2. After reindex, next_poll_ns resets to now + interval_ms
3. Rapid-fire touch calls can't cause thrashing — at most one immediate follow-up check after a reindex

Proposed fix

1. Option A — append --recurse-submodules to git_is_dirty() in src/watcher/watcher.c (one-line change)
2. Option C — expose cbm_watcher_touch() as an MCP tool so hooks/scripts can trigger immediate reindex (~30-50 LOC)

Happy to open a PR for either or both if this approach sounds right.