Skip to content

feat(opencode): add db stats, prune, and vacuum CLI commands - #43456

Closed
AndyS77 wants to merge 4 commits into
anomalyco:devfrom
AndyS77:db-cleanup
Closed

feat(opencode): add db stats, prune, and vacuum CLI commands#43456
AndyS77 wants to merge 4 commits into
anomalyco:devfrom
AndyS77:db-cleanup

Conversation

@AndyS77

@AndyS77 AndyS77 commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown

Issue for this PR

Partially addresses #33356

Type of change

  • Bug fix
  • Refactor / code improvement

What does this PR do?

VACUUM INTO + file swap (replaces plain VACUUM):

Plain VACUUM in WAL mode can write a file comparable to the database size into the WAL before completing. On a 104 GB database, VACUUM wrote ~56 GB into the WAL within 45 minutes without completing (see #33356 comment).

The db vacuum command now uses a two-phase approach:

  1. PRAGMA wal_checkpoint(TRUNCATE) — flush WAL into main database
  2. VACUUM INTO '<path>.vacuum' — create compacted copy to separate file
  3. Close connection, atomic file swap, clean up WAL/SHM files
  4. Reopen and verify with page count comparison

Retention-based pruning (--max-age <days>):

Orphan-only pruning reclaims almost nothing on real-world databases where bloat is dominated by events for sessions users still have in their list. After deleting all sessions older than 30 days on the 104 GB database, 85 GB of events remained for live sessions.

The db prune command now supports --max-age <days> to delete sessions older than N days (based on time_updated), then cascades through pruneOrphanedEvents to clean up their events.

Removed --vacuum flag from prune: Running VACUUM inside the prune handler was unsafe (same WAL blow-up risk). Users should run opencode db vacuum as a separate step after pruning.

How did you verify your code works?

  • dbStats test: verifies table row counts, page count, page size, freelist count
  • pruneOrphanedEvents tests: verifies orphaned events are deleted, live session events are preserved
  • pruneOldSessions tests: verifies sessions older than max-age are deleted with their events, recent sessions are preserved
  • bun turbo typecheck --concurrency=1 --filter=opencode: passes clean
  • Code reviewed by @code-reviewer (previous commits)

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

AndyS77 and others added 3 commits August 19, 2026 16:36
@AndyS77 zai-glm-52
test(opencode): db maintenance CLI tests - stats, prune, vacuum
a091662 
Refs BUG-33356

Co-Authored-By: zai-glm-52 <noreply@ai.local>
Agent: @Feature-dev
Scope: BUG-33356
@AndyS77
feat(opencode): add db stats, prune, and vacuum CLI commands
319f6c8 
- db stats: shows table row counts, database size, freelist pages
- db prune: removes orphaned events for deleted sessions (--dry-run, --vacuum)
- db vacuum: reclaims free space with VACUUM and reports reclaimed MB
- auto_vacuum = INCREMENTAL enabled for new databases
- Exports dbStats() and pruneOrphanedEvents() for programmatic use

Refs BUG-33356
@AndyS77
fix(opencode): address code review findings for db maintenance
90b029e 
- Fix pruneOrphanedEvents: use same orphan criteria for event_sequence
  deletion (LEFT JOIN session) instead of NOT IN (SELECT FROM event)
  which could delete valid sequence entries for live sessions
- Fix VacuumCommand: use PRAGMA page_size instead of hardcoded 4096
- Remove unnecessary @opentelemetry/sdk-trace-web dev dependency

Refs BUG-33356
@github-actions github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Aug 19, 2026
@AndyS77 AndyS77 mentioned this pull request Aug 19, 2026
Open
@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Thanks for updating your PR! It now meets our contributing guidelines. 👍

renekris added a commit to renekris/opencode-lowmem that referenced this pull request Aug 26, 2026
@renekris
feat(opencode): db stats and vacuum subcommands (port of upstream ano…
ea44d23 
…malyco#43456, partial)

`opencode db stats` reports page metadata, freelist, db/WAL sizes and
per-table row counts with approximate payload bytes over a read-only
connection (no AppRuntime — its shared layer opens the db writable).
`opencode db vacuum` is an offline explicit VACUUM behind a liveness
guard: on Linux a best-effort /proc scan for open db/WAL/SHM handles
plus an exclusive lock; elsewhere the exclusive lock only, with the
reduced guarantee printed. locking_mode=EXCLUSIVE is taken before the
guard transaction so the lock persists through COMMIT into VACUUM.
Not ported from anomalyco#43456: orphan prune, auto_vacuum.

Upstream anomalyco#43456 by AndyS77; adapted (low-level wrapper, lock
sequencing, proofs) for the fork.

Tests: stats/vacuum on real SQLite fixtures including WAL lock-interval
regression, live-process refusal, and reclaim-on-bloat.

README: ported row for anomalyco#43456 (partial); customs rows for durable
paging and the background-job terminal ring.
renekris added a commit to renekris/opencode-lowmem that referenced this pull request Aug 26, 2026
@renekris
chore(build): hard-fail on shim mismatch; overhaul fork README
1a54eb5 
fork-build.sh now exits 1 when the live shim does not point at the fresh
binary unless OPENCODE_FORK_BUILD_ALLOW_SHIM_MISMATCH=1 is set, and asserts
the shim reports the stamped version after install. README gains the
anomalyco#43456 partial-port row (AndyS77), durable-paging and terminal-ring
customs rows, the complete 17-knob inventory, the v1.18.22 base, corrected
author credits, and an honest deferred-ports section (anomalyco#40698 is deferred
for scope, not a missing seam).
@romanilyin romanilyin mentioned this pull request Aug 28, 2026
Open
4 of 6 tasks
@heisenberg600 heisenberg600 mentioned this pull request Sep 6, 2026
Open
3 of 6 tasks
@Yuepixel Yuepixel mentioned this pull request Sep 10, 2026
Open
@AndyS77 opencode
fix(opencode): use VACUUM INTO + file swap and add --max-age pruning
37b0d71 
Plain VACUUM in WAL mode can write a file comparable to the database size
into the WAL before completing. On large databases (100+ GB) this causes
the WAL to blow up and the operation to never finish.

Replace plain VACUUM with VACUUM INTO + atomic file swap:
  1. PRAGMA wal_checkpoint(TRUNCATE) — flush WAL into main DB
  2. VACUUM INTO '<path>.vacuum' — create compacted copy
  3. Close connection, swap files, clean up WAL/SHM

Add --max-age <days> option to 'db prune' for retention-based pruning of
live sessions. Orphan-only pruning reclaims almost nothing on real-world
databases where bloat is dominated by events for sessions users still have
in their list (anomalyco#33356).

Remove --vacuum flag from prune (separate 'db vacuum' command is safer).

Refs anomalyco#33356

Co-Authored-By: opencode <opencode@anomaly.co>
Agent: @bug-fix
Model: msp-crow/zai-glm-52
Scope: anomalyco#33356
@AndyS77

AndyS77 commented Sep 16, 2026

Copy link
Copy Markdown
Author

Updated based on community data from #33356 (#33356 (comment)):

1. VACUUM INTO + file swap — plain VACUUM in WAL mode causes WAL blow-up on large databases. Now uses VACUUM INTO to a separate file, then atomic swap after closing the connection.

2. --max-age pruning — orphan-only pruning reclaims almost nothing on real-world databases where bloat is dominated by events for live sessions. Added --max-age option to prune sessions older than N days.

3. Removed --vacuum from prune — VACUUM inside the prune handler was unsafe. Users should run 'opencode db vacuum' separately.

Also related: #41711 (feat(storage): add safe database maintenance controls) is a more comprehensive approach that explicitly references this PR as complementary.

@github-actions

github-actions Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Automated PR Cleanup

Thank you for contributing to opencode.

Due to the high volume of PRs from users and AI agents, we periodically close older PRs using automated criteria so maintainers can focus review time on the most active and community-supported contributions.

This PR was closed because it matched the following cleanup criteria:

  • The PR was created more than 1 month ago
  • The PR had fewer than 2 positive reactions
  • Positive reactions are counted as thumbs-up, heart, celebration, or rocket reactions on the PR

PRs created within the last month are not affected by this cleanup.

If you believe this PR was closed incorrectly, or if you are still actively working on it, please leave a comment explaining why it should be reopened. A maintainer can review and reopen it if appropriate.

Thanks again for taking the time to contribute.

@github-actions github-actions Bot closed this Sep 19, 2026
@github-actions github-actions Bot mentioned this pull request Sep 19, 2026
Open
@DScoNOIZ DScoNOIZ mentioned this pull request Sep 20, 2026
Open
@AndyS77

AndyS77 commented Sep 21, 2026

Copy link
Copy Markdown
Author

This PR was closed by automated PR cleanup (older than 1 month, fewer than 2 reactions). I'd like to request reopening because:

The underlying issue (#33356) is still open and affects a large number of users. The issue has 5,700+ open issues/PRs referencing it, with users reporting databases reaching 50–104 GB. The event table grows unboundedly with no built-in mechanism to reclaim space.

This PR was recently updated with community-informed fixes. Based on data from #33356 comment (104 GB database on Windows), the PR now includes:

  • VACUUM INTO + atomic file swap (instead of plain VACUUM which causes WAL blow-up on large databases)
  • --max-age <days> retention pruning for live sessions (orphan-only pruning reclaims almost nothing on real-world databases where 85 GB of events belong to sessions users still have)
  • Separate db vacuum command (removed unsafe --vacuum flag from prune)

No merged alternative exists. While #41711 (feat(storage): add safe database maintenance controls) is a more comprehensive approach, it has also been open since August without being merged. This PR provides immediate relief with minimal, well-tested CLI commands.

The PR is complementary, not competing. #41711 explicitly references this PR and notes that "shared CLI/status pieces can be reconciled during review if #43456 lands first."

I'm still actively maintaining this PR and would appreciate it being reopened.

@AndyS77

AndyS77 commented Sep 29, 2026

Copy link
Copy Markdown
Author

Rebased on latest origin/dev and force-pushed to my fork: https://github.com/AndyS77/opencode/commits/db-cleanup

What this adds

opencode db prune --max-age 168 deletes events for idle sessions (sessions, messages, and parts stay intact and readable). --purge-sessions also deletes the sessions themselves. The newest session is always exempted so the active workspace is never touched.

opencode db vacuum uses VACUUM INTO + file swap instead of plain VACUUM to avoid WAL blow-up on large databases.

This implements the event-pruning policy validated in #33356 (comment) — deleting events for idle sessions (not the sessions) is what bounds the table, and the durable state in message/part keeps everything readable.

This PR was closed by automated cleanup (>1 month, <2 reactions). Requesting reopen — the implementation is complete and ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

automated-pr-cleanup

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AndyS77