diff --git a/apps/loopover-ui/content/docs/ams-operations-runbook.mdx b/apps/loopover-ui/content/docs/ams-operations-runbook.mdx
new file mode 100644
index 0000000000..bf4cbea18f
--- /dev/null
+++ b/apps/loopover-ui/content/docs/ams-operations-runbook.mdx
@@ -0,0 +1,280 @@
+---
+title: AMS operations runbook
+description: Recover from SQLite lock contention, ledger corruption, and post-upgrade schema migrations in loopover-miner's local state.
+---
+
+Operator-facing runbook for **local SQLite state**: what the concurrency guarantees actually mean,
+how to recover from corruption, what to do when two miner processes collide on the same files, and
+how schema upgrades migrate your on-disk ledgers after a package update.
+
+
+ Scope: AMS local stores only. For laptop/fleet deployment layout see [AMS deployment
+ guide](/docs/ams-deployment). This runbook does **not** cover the self-hosted **review stack**
+ (Orb/API/LoopOver DB).
+
+
+## Local state at a glance
+
+Every miner keeps **independent SQLite files** under one state directory (default
+`~/.config/loopover-miner/`, override with `LOOPOVER_MINER_CONFIG_DIR`). Each store has its own
+file, table, and optional per-store env override. Files are created with **`0700` directories /
+`0600` database files** on first open.
+
+Common files you will touch in incidents:
+
+- `laptop-state.sqlite3` — bootstrap metadata (`loopover-miner init`)
+- `claim-ledger.sqlite3` — soft issue claims on this machine
+- `event-ledger.sqlite3` — append-only manage-loop audit trail
+- `portfolio-queue.sqlite3` — per-repo portfolio queue
+- `run-state.sqlite3` — discover/plan/prepare phase markers
+- `attempt-log.sqlite3` — per-attempt coding-agent driver events
+- `prediction-ledger.sqlite3` — predicted gate verdicts for self-improve
+- `plan-store.sqlite3` — persisted MCP plan DAGs
+- `governor-ledger.sqlite3` — governor allow/deny/throttle decisions
+
+## SQLite concurrency — what busy_timeout guarantees
+
+Every store opened through the miner's local-store layer sets:
+
+
+
+Default **5000 ms**; overridable per-test only — production stores always use the default.
+
+
+
+
+ **Invariant:** one active loop (or one intentional writer set) per state directory. Horizontal
+ scale = **isolated state dirs** — separate compose projects, separate
+ `LOOPOVER_MINER_CONFIG_DIR`, or the Kubernetes StatefulSet pattern in the [AMS deployment
+ guide](/docs/ams-deployment).
+
+
+Quick health check — `doctor` includes `laptop-state-sqlite` (file exists + readable) and
+`state-dir-writable`, and performs **no network I/O**:
+
+
+
+## Scenario: two miners collided
+
+**Symptoms:** `database is locked` / `SQLITE_BUSY` in logs or stderr; duplicate or out-of-order
+event sequences after an unclean shutdown; two systemd units, two `docker compose --scale
+miner=N` replicas, or a manual `loop` plus a supervised `loop` sharing one config dir; claims or
+queue rows flipping unexpectedly.
+
+**Diagnosis.** List processes using the state dir, confirm only one long-lived miner should own
+it, then inspect soft claims and the queue without mutating anything:
+
+/dev/null || fuser -v "$STATE_DIR"/*.sqlite3 2>/dev/null
+
+loopover-miner claim list --json
+loopover-miner queue list --json
+loopover-miner ledger list --json | tail -20`}
+/>
+
+**Remediation.** Stop all but one miner process targeting that state dir (`systemctl stop`,
+`docker compose down`, kill stray `loop`). For N parallel workers, give each an isolated state
+path instead of sharing one volume:
+
+
+
+Re-run `loopover-miner doctor`. If locks persist with a single process, see **Ledger corrupted**
+below.
+
+
+ Claims are local bookkeeping only. Two miners on different machines claiming the same GitHub
+ issue is a **fleet coordination** problem (duplicate-cluster adjudication in the engine), not
+ something SQLite resolves — split state dirs and use operational claim hygiene.
+
+
+## Backup and restore
+
+Proactive tooling, not just the reactive "ledger corrupted" scenario below — run
+`backup-miner.sh` on a schedule (cron, systemd timer, etc.) so a good restore point always exists
+before anything goes wrong.
+
+`scripts/backup-miner.sh` backs up every `*.sqlite3` file currently present under
+`LOOPOVER_MINER_CONFIG_DIR` into a new timestamped directory, using SQLite's own online `.backup`
+command (safe even while the miner is running) plus a `PRAGMA integrity_check` on each resulting
+file before it's kept. Stores are discovered by glob, not a hardcoded list, so a newly added store
+is backed up automatically.
+
+
+
+`scripts/restore-miner.sh` is the read side. **Stop the miner first** (it does not detect a
+running process). It validates every store file in the chosen backup with `PRAGMA
+integrity_check` **before** copying anything into place — a half-good backup can never produce a
+half-restored state directory. Requires an explicit `--yes` flag and defaults to the newest
+backup when no directory is given:
+
+ # a specific one
+loopover-miner doctor --json # verify afterward`}
+/>
+
+It also removes any leftover `-wal`/`-shm` sidecar files from the live directory after restoring
+each store — those hold in-flight writes from *before* the restore, and leaving them in place
+would let SQLite silently replay stale pre-restore writes back on top of the freshly restored
+file on next open.
+
+## Scenario: ledger corrupted
+
+**Symptoms:** a command throws `corrupted_*_row` (`corrupted_attempt_log_row`,
+`corrupted_governor_row`, `corrupted_plan_row`, `corrupted_prediction_row`, …); `loopover-miner
+doctor` reports `laptop-state-sqlite` not readable; `sqlite3` reports `database disk image is
+malformed`; partial writes after disk full, forced kill during a migration transaction, or
+copying a live `.sqlite3` while the miner is writing.
+
+**Diagnosis.** Identify which file fails, then run a read-only probe and check the filesystem:
+
+
+
+Check disk space, permissions (`0600` file, `0700` parent), and backup tools copying mid-write.
+
+**Remediation.** Stop the miner, then back up the whole state directory (even damaged files help
+post-mortems):
+
+
+
+Choose a recovery tier:
+
+ sh scripts/restore-miner.sh --yes -> restart.",
+ },
+ {
+ title: "C — full re-init",
+ description:
+ "Multiple files suspect or state is disposable. Archive the dir -> loopover-miner init -> reconfigure env/goals. Rebuild claims/plans from GitHub metadata as needed.",
+ },
+ ]}
+/>
+
+
+ Never copy a live SQLite file from a running miner as backup — stop first, or use SQLite's own
+ `.backup` command: {`sqlite3 "$DB" ".backup '\${DB}.safe-copy'"`}
+
+
+After recovery, run `loopover-miner doctor --json` and spot-check read-only listings (`claim
+list`, `ledger list`). Append-only stores **do not repair individual bad rows** in place —
+corrupted payload JSON is rejected on read by design so bad data cannot silently propagate.
+
+## Scenario: migrate ledgers after a package upgrade
+
+Stores use a lightweight schema-version convention: bootstrap `CREATE TABLE IF NOT EXISTS …` is
+schema **version 1**; each store may register post-baseline migrations that run on every open;
+the version is stamped in SQLite's **`PRAGMA user_version`**; migrations run **once**, in order,
+inside a transaction, and a failed migration rolls back and retries on next open.
+**Downgrade is not supported** — older miner versions may not read files written by newer
+migrations.
+
+**Before upgrading** the `@loopover/miner` package (npm, image tag, or git pull), snapshot state:
+
+ /tmp/miner-pre-upgrade-doctor.json
+STATE_DIR="$(loopover-miner status --json | jq -r .stateDir)"
+tar -czf "/tmp/loopover-miner-state-$(date +%Y%m%d).tar.gz" -C "$(dirname "$STATE_DIR")" "$(basename "$STATE_DIR")"`}
+/>
+
+**Stop** supervised loops (`systemctl stop loopover-miner.service`, `docker compose stop
+miner`, etc.), then **install** the new version. The CLI prints a one-line npm upgrade nudge when
+behind registry latest — informational only.
+
+**Migrate**, before starting any miner process, so every existing store is brought up to date in
+one deliberate pass instead of relying on whichever command happens to open a given store first:
+
+
+
+
+ Pending migrations still apply automatically on first open regardless — `migrate` is the
+ proactive, explicit alternative to waiting for that implicit path. A store file that hasn't been
+ created yet is reported as skipped, not created; `migrate` never bootstraps fresh state.
+
+
+**Start** one miner process, then **verify**:
+
+
+
+If a migration throws on startup, **do not delete files immediately** — restore the pre-upgrade
+tarball, pin the previous package version, and file an issue with the failing `user_version` and
+store filename.
+
+
+ Upgrade and restart **one worker/state dir at a time** so isolated workers never share a
+ directory mid-migration.
+
+
+## Related docs
+
+- [AMS deployment guide](/docs/ams-deployment) — laptop vs fleet, volumes, systemd, scaling rules
+- [`packages/loopover-miner/README.md`](https://github.com/JSONbored/loopover/blob/main/packages/loopover-miner/README.md#local-storage) — store inventory
+- [`packages/loopover-miner/docs/env-reference.md`](https://github.com/JSONbored/loopover/blob/main/packages/loopover-miner/docs/env-reference.md) — per-store path overrides
+- [`packages/loopover-miner/docs/coding-agent-driver.md`](https://github.com/JSONbored/loopover/blob/main/packages/loopover-miner/docs/coding-agent-driver.md) — attempt log semantics
+- [`packages/loopover-miner/docs/discovery-plane-operator-guide.md`](https://github.com/JSONbored/loopover/blob/main/packages/loopover-miner/docs/discovery-plane-operator-guide.md) — optional hosted plane (distinct from local ledger ops)
diff --git a/apps/loopover-ui/src/components/site/docs-nav.tsx b/apps/loopover-ui/src/components/site/docs-nav.tsx
index 97404bd7cd..b9948a5689 100644
--- a/apps/loopover-ui/src/components/site/docs-nav.tsx
+++ b/apps/loopover-ui/src/components/site/docs-nav.tsx
@@ -74,7 +74,10 @@ export const docsNav: DocsGroup[] = [
},
{
title: "AMS: deployment",
- items: [{ to: "/docs/ams-deployment", label: "Deployment guide" }],
+ items: [
+ { to: "/docs/ams-deployment", label: "Deployment guide" },
+ { to: "/docs/ams-operations-runbook", label: "Operations runbook" },
+ ],
},
],
},
diff --git a/apps/loopover-ui/src/routeTree.gen.ts b/apps/loopover-ui/src/routeTree.gen.ts
index b7f3143710..68858ebb2c 100644
--- a/apps/loopover-ui/src/routeTree.gen.ts
+++ b/apps/loopover-ui/src/routeTree.gen.ts
@@ -57,6 +57,7 @@ import { Route as DocsGithubAppRouteImport } from './routes/docs.github-app'
import { Route as DocsFumadocsSpikeApiReferenceRouteImport } from './routes/docs.fumadocs-spike-api-reference'
import { Route as DocsBranchAnalysisRouteImport } from './routes/docs.branch-analysis'
import { Route as DocsBetaOnboardingRouteImport } from './routes/docs.beta-onboarding'
+import { Route as DocsAmsOperationsRunbookRouteImport } from './routes/docs.ams-operations-runbook'
import { Route as DocsAmsDeploymentRouteImport } from './routes/docs.ams-deployment'
import { Route as DocsAiSummariesRouteImport } from './routes/docs.ai-summaries'
import { Route as AppWorkbenchRouteImport } from './routes/app.workbench'
@@ -329,6 +330,12 @@ const DocsBetaOnboardingRoute = DocsBetaOnboardingRouteImport.update({
path: '/beta-onboarding',
getParentRoute: () => DocsRoute,
} as any)
+const DocsAmsOperationsRunbookRoute =
+ DocsAmsOperationsRunbookRouteImport.update({
+ id: '/ams-operations-runbook',
+ path: '/ams-operations-runbook',
+ getParentRoute: () => DocsRoute,
+ } as any)
const DocsAmsDeploymentRoute = DocsAmsDeploymentRouteImport.update({
id: '/ams-deployment',
path: '/ams-deployment',
@@ -442,6 +449,7 @@ export interface FileRoutesByFullPath {
'/app/workbench': typeof AppWorkbenchRoute
'/docs/ai-summaries': typeof DocsAiSummariesRoute
'/docs/ams-deployment': typeof DocsAmsDeploymentRoute
+ '/docs/ams-operations-runbook': typeof DocsAmsOperationsRunbookRoute
'/docs/beta-onboarding': typeof DocsBetaOnboardingRoute
'/docs/branch-analysis': typeof DocsBranchAnalysisRoute
'/docs/fumadocs-spike-api-reference': typeof DocsFumadocsSpikeApiReferenceRoute
@@ -506,6 +514,7 @@ export interface FileRoutesByTo {
'/app/workbench': typeof AppWorkbenchRoute
'/docs/ai-summaries': typeof DocsAiSummariesRoute
'/docs/ams-deployment': typeof DocsAmsDeploymentRoute
+ '/docs/ams-operations-runbook': typeof DocsAmsOperationsRunbookRoute
'/docs/beta-onboarding': typeof DocsBetaOnboardingRoute
'/docs/branch-analysis': typeof DocsBranchAnalysisRoute
'/docs/fumadocs-spike-api-reference': typeof DocsFumadocsSpikeApiReferenceRoute
@@ -574,6 +583,7 @@ export interface FileRoutesById {
'/app/workbench': typeof AppWorkbenchRoute
'/docs/ai-summaries': typeof DocsAiSummariesRoute
'/docs/ams-deployment': typeof DocsAmsDeploymentRoute
+ '/docs/ams-operations-runbook': typeof DocsAmsOperationsRunbookRoute
'/docs/beta-onboarding': typeof DocsBetaOnboardingRoute
'/docs/branch-analysis': typeof DocsBranchAnalysisRoute
'/docs/fumadocs-spike-api-reference': typeof DocsFumadocsSpikeApiReferenceRoute
@@ -643,6 +653,7 @@ export interface FileRouteTypes {
| '/app/workbench'
| '/docs/ai-summaries'
| '/docs/ams-deployment'
+ | '/docs/ams-operations-runbook'
| '/docs/beta-onboarding'
| '/docs/branch-analysis'
| '/docs/fumadocs-spike-api-reference'
@@ -707,6 +718,7 @@ export interface FileRouteTypes {
| '/app/workbench'
| '/docs/ai-summaries'
| '/docs/ams-deployment'
+ | '/docs/ams-operations-runbook'
| '/docs/beta-onboarding'
| '/docs/branch-analysis'
| '/docs/fumadocs-spike-api-reference'
@@ -774,6 +786,7 @@ export interface FileRouteTypes {
| '/app/workbench'
| '/docs/ai-summaries'
| '/docs/ams-deployment'
+ | '/docs/ams-operations-runbook'
| '/docs/beta-onboarding'
| '/docs/branch-analysis'
| '/docs/fumadocs-spike-api-reference'
@@ -1167,6 +1180,13 @@ declare module '@tanstack/react-router' {
preLoaderRoute: typeof DocsBetaOnboardingRouteImport
parentRoute: typeof DocsRoute
}
+ '/docs/ams-operations-runbook': {
+ id: '/docs/ams-operations-runbook'
+ path: '/ams-operations-runbook'
+ fullPath: '/docs/ams-operations-runbook'
+ preLoaderRoute: typeof DocsAmsOperationsRunbookRouteImport
+ parentRoute: typeof DocsRoute
+ }
'/docs/ams-deployment': {
id: '/docs/ams-deployment'
path: '/ams-deployment'
@@ -1340,6 +1360,7 @@ const AppRouteWithChildren = AppRoute._addFileChildren(AppRouteChildren)
interface DocsRouteChildren {
DocsAiSummariesRoute: typeof DocsAiSummariesRoute
DocsAmsDeploymentRoute: typeof DocsAmsDeploymentRoute
+ DocsAmsOperationsRunbookRoute: typeof DocsAmsOperationsRunbookRoute
DocsBetaOnboardingRoute: typeof DocsBetaOnboardingRoute
DocsBranchAnalysisRoute: typeof DocsBranchAnalysisRoute
DocsFumadocsSpikeApiReferenceRoute: typeof DocsFumadocsSpikeApiReferenceRoute
@@ -1381,6 +1402,7 @@ interface DocsRouteChildren {
const DocsRouteChildren: DocsRouteChildren = {
DocsAiSummariesRoute: DocsAiSummariesRoute,
DocsAmsDeploymentRoute: DocsAmsDeploymentRoute,
+ DocsAmsOperationsRunbookRoute: DocsAmsOperationsRunbookRoute,
DocsBetaOnboardingRoute: DocsBetaOnboardingRoute,
DocsBranchAnalysisRoute: DocsBranchAnalysisRoute,
DocsFumadocsSpikeApiReferenceRoute: DocsFumadocsSpikeApiReferenceRoute,
diff --git a/apps/loopover-ui/src/routes/docs.ams-operations-runbook.tsx b/apps/loopover-ui/src/routes/docs.ams-operations-runbook.tsx
new file mode 100644
index 0000000000..ca82473ec8
--- /dev/null
+++ b/apps/loopover-ui/src/routes/docs.ams-operations-runbook.tsx
@@ -0,0 +1,49 @@
+import { createFileRoute, notFound } from "@tanstack/react-router";
+import { Suspense } from "react";
+
+import { DocsPage } from "@/components/site/docs-page";
+import { docsClientLoader } from "@/lib/docs-client-loader";
+
+// Rendered from content/docs/ams-operations-runbook.mdx via fumadocs-mdx's browser entry
+// (docsClientLoader), through the existing DocsPage/Callout/CodeBlock/FeatureRow
+// primitives -- not fumadocs-ui's bundled components. See docs-source.ts's comment
+// for why the loader below resolves only a plain, serializable path string.
+export const Route = createFileRoute("/docs/ams-operations-runbook")({
+ loader: async () => {
+ const { docsSource } = await import("@/lib/docs-source");
+ const page = docsSource.getPage(["ams-operations-runbook"]);
+ if (!page) throw notFound();
+ return { path: page.path, title: page.data.title, description: page.data.description };
+ },
+ head: () => ({
+ meta: [
+ { title: "AMS operations runbook — LoopOver docs" },
+ {
+ name: "description",
+ content:
+ "Recover from SQLite lock contention, ledger corruption, and post-upgrade schema migrations in loopover-miner's local state.",
+ },
+ { property: "og:title", content: "AMS operations runbook — LoopOver docs" },
+ {
+ property: "og:description",
+ content:
+ "Recover from SQLite lock contention, ledger corruption, and post-upgrade schema migrations in loopover-miner's local state.",
+ },
+ { property: "og:url", content: "/docs/ams-operations-runbook" },
+ ],
+ links: [{ rel: "canonical", href: "/docs/ams-operations-runbook" }],
+ }),
+ component: AmsOperationsRunbook,
+});
+
+function AmsOperationsRunbook() {
+ const { path, title, description } = Route.useLoaderData();
+ const Content = docsClientLoader.getComponent(path);
+ return (
+
+ Loading…
}>
+
+
+
+ );
+}
diff --git a/apps/loopover-ui/src/routes/docs.index.tsx b/apps/loopover-ui/src/routes/docs.index.tsx
index 1eb9096880..e93aa662c4 100644
--- a/apps/loopover-ui/src/routes/docs.index.tsx
+++ b/apps/loopover-ui/src/routes/docs.index.tsx
@@ -74,6 +74,7 @@ const AUDIENCES: Audience[] = [
{ to: "/docs/maintainer-self-hosting", label: "Self-host reviews" },
{ to: "/docs/self-hosting-unified-ams-orb", label: "Unified ORB + AMS" },
{ to: "/docs/ams-deployment", label: "AMS deployment guide" },
+ { to: "/docs/ams-operations-runbook", label: "AMS operations runbook" },
{ to: "/docs/self-hosting-docs-audit", label: "Self-host docs audit" },
{ to: "/docs/maintainer-install-trust", label: "Install & trust guide" },
{ to: "/docs/github-app", label: "GitHub App configuration" },
diff --git a/packages/loopover-miner/docs/operations-runbook.md b/packages/loopover-miner/docs/operations-runbook.md
index aac515afb4..9a43bfdae8 100644
--- a/packages/loopover-miner/docs/operations-runbook.md
+++ b/packages/loopover-miner/docs/operations-runbook.md
@@ -1,5 +1,9 @@
# loopover-miner — operational runbook
+> Also published on the docs website: [AMS operations runbook](https://loopover.ai/docs/ams-operations-runbook)
+> (same content, rendered with search and the rest of the maintainer docs nav). This file remains
+> the canonical source and ships inside the published `@loopover/miner` package.
+
Operator-facing runbook for **local SQLite state**: what the concurrency guarantees actually mean, how to recover from corruption, what to do when two miner processes collide on the same files, and how schema upgrades migrate your on-disk ledgers after a package update.
> **Scope:** AMS local stores only. For laptop/fleet deployment layout see [`../DEPLOYMENT.md`](../DEPLOYMENT.md). For Grafana setup see [#5190](https://github.com/JSONbored/gittensory/issues/5190). For the optional hosted discovery plane see [`discovery-plane-operator-guide.md`](discovery-plane-operator-guide.md). This runbook does **not** cover the self-hosted **review stack** (Orb/API/LoopoverDB).