From a032c9a17e3755cee2e4275f485995affcbcf8ec Mon Sep 17 00:00:00 2001 From: jsdevninja Date: Wed, 15 Jul 2026 01:43:16 -0500 Subject: [PATCH] docs(selfhost): add combined ORB + AMS quickstart Document standing up root docker-compose.yml with AMS fleet/laptop on one host, including LOOPOVER_MINER_CONFIG_DIR agreement and the fleet ams-observability bridge. Closes #5996. Co-authored-by: Cursor --- ORB_AMS_QUICKSTART.md | 147 ++++++++++++++++++++++++++ README.md | 7 +- packages/loopover-miner/DEPLOYMENT.md | 4 +- packages/loopover-miner/README.md | 2 +- 4 files changed, 157 insertions(+), 3 deletions(-) create mode 100644 ORB_AMS_QUICKSTART.md diff --git a/ORB_AMS_QUICKSTART.md b/ORB_AMS_QUICKSTART.md new file mode 100644 index 0000000000..f3bb06b802 --- /dev/null +++ b/ORB_AMS_QUICKSTART.md @@ -0,0 +1,147 @@ +# Running ORB + AMS together + +Operator quickstart for self-hosting **ORB** (the review stack in the root [`docker-compose.yml`](docker-compose.yml)) and **AMS** (`@loopover/miner`, fleet or laptop) on **one host**, including the Grafana `ams-observability` bridge. + +Each product works alone. This page is the single path that documents how their compose files, volumes, and `LOOPOVER_MINER_CONFIG_DIR` interact. AMS-only deploy details stay in [`packages/loopover-miner/DEPLOYMENT.md`](packages/loopover-miner/DEPLOYMENT.md); AMS Grafana wiring stays in [`packages/loopover-miner/docs/observability.md`](packages/loopover-miner/docs/observability.md). + +## What you get + +| Piece | Compose file | Role | +| ----- | ------------ | ---- | +| ORB | root `docker-compose.yml` | Self-hosted LoopOver API / review agent (and optional Prometheus + Grafana) | +| AMS fleet worker | `packages/loopover-miner/docker-compose.miner.yml` | Long-lived miner container (`command: ["run"]`) | +| AMS → Grafana bridge | same root compose, `--profile ams-observability` | `ams-reporting-exporter` redacts miner ledgers into Grafana's reporting volume | + +Grafana itself lives under `--profile observability`. The AMS exporter is a **separate** profile (`ams-observability`) so engine-only deployments do not start it. For panels to appear you need **both** profiles (plus a miner writing ledgers the exporter can read). + +## Prerequisites + +1. Docker Compose v2 on the host. +2. ORB secrets: copy [`.env.selfhost.example`](.env.selfhost.example) (or [`.env.example`](.env.example)) → `.env` at the repo root and fill GitHub App credentials. Never commit real values. +3. AMS secrets (fleet path): from `packages/loopover-miner/`: + + ```sh + cp .loopover-miner.env.example .loopover-miner.env + # fill GITHUB_TOKEN (+ optional coding-agent provider keys) + ``` + +4. Optional — custom shared state directory (defaults are fine): + + ```sh + # In the repo-root `.env` (and exported in the shell if you prefer): + LOOPOVER_MINER_CONFIG_DIR=~/.config/loopover-miner + ``` + +## Env vars that must agree + +| Variable | ORB side | AMS side | Notes | +| -------- | -------- | -------- | ----- | +| `LOOPOVER_MINER_CONFIG_DIR` | Host path mounted **read-only** on `ams-reporting-exporter` at `/ams-ledgers` (default `~/.config/loopover-miner`) | **Laptop:** where the CLI writes SQLite. **Fleet + override:** host path bind-mounted at the container's `/data/miner` | Set once; both sides use the same default when unset. | +| AMS credentials | Not shared with ORB | `.loopover-miner.env` (`GITHUB_TOKEN`, …) | ORB uses the root `.env` GitHub App material; do not mix the two secret files. | + +### Laptop mode vs fleet mode state + +| AMS mode | Where ledgers live by default | Extra step for `ams-observability` | +| -------- | ----------------------------- | ---------------------------------- | +| Laptop (`loopover-miner` on the host) | `~/.config/loopover-miner` | None — matches the exporter bind mount | +| Fleet (`docker-compose.miner.yml` alone) | Named Docker volume `miner-data` → `/data/miner` | **Required:** copy the opt-in override so fleet state is a **host** directory the exporter can read (see below). Without it, Grafana AMS datasources stay **silently empty**. | + +The fleet bridge is the `#5805` override in [`packages/loopover-miner/docker-compose.miner.override.yml.example`](packages/loopover-miner/docker-compose.miner.override.yml.example). It replaces the named-volume mount for `/data/miner` with `${LOOPOVER_MINER_CONFIG_DIR:-~/.config/loopover-miner}:/data/miner` — the same expression `ams-reporting-exporter` already uses. + +## Worked example: ORB + AMS fleet + Grafana AMS panels + +Run from the **monorepo root**. This is the verified two-product invocation (ORB profiles + AMS fleet compose + fleet↔exporter bridge). + +```sh +# 1) One-time: opt-in host bind so fleet state matches the exporter (gitignored after copy) +cp packages/loopover-miner/docker-compose.miner.override.yml.example \ + packages/loopover-miner/docker-compose.miner.override.yml + +# 2) Stand up ORB (API + observability stack) and AMS exporter + fleet miner together +docker compose \ + -f docker-compose.yml \ + -f packages/loopover-miner/docker-compose.miner.yml \ + -f packages/loopover-miner/docker-compose.miner.override.yml \ + --profile observability \ + --profile ams-observability \ + up -d --build +``` + +### Expected output (shape) + +`docker compose … up -d --build` should finish with services including at least: + +- ORB core (e.g. `loopover`, Redis / SQLite stack depending on your base profiles) +- `grafana`, `prometheus`, … from `--profile observability` +- `ams-reporting-exporter` from `--profile ams-observability` +- `miner` from `docker-compose.miner.yml` + +Then: + +```sh +docker compose \ + -f docker-compose.yml \ + -f packages/loopover-miner/docker-compose.miner.yml \ + -f packages/loopover-miner/docker-compose.miner.override.yml \ + --profile observability \ + --profile ams-observability \ + ps +``` + +`ams-reporting-exporter` and `miner` should be `running`. After the miner has written `attempt-log.sqlite3` / `prediction-ledger.sqlite3` under `LOOPOVER_MINER_CONFIG_DIR`, the exporter's healthcheck expects a non-empty redacted `/reporting/ams-attempt-log.sqlite` (default interval 30s). + +### Confirm Grafana AMS panels (fleet mode) + +1. Open Grafana (default published port from the observability profile — see your `.env` / compose port mapping). +2. **Connections → Data sources** should list **AMS Attempt Log** and **AMS Prediction Ledger** (provisioned from [`grafana/provisioning/datasources/ams-ledgers.yml`](grafana/provisioning/datasources/ams-ledgers.yml)). +3. Explore either datasource after at least one export cycle. Empty tables before the miner has logged attempts is normal; **empty forever** usually means the exporter and miner still disagree on the host path (missing override, or a custom `LOOPOVER_MINER_CONFIG_DIR` set on only one side). + +Deep dive on what is exported vs kept private: [`packages/loopover-miner/docs/observability.md`](packages/loopover-miner/docs/observability.md). Bridge rationale and AMS-only wording: [DEPLOYMENT.md — Running fleet mode alongside ORB's `ams-observability` profile](packages/loopover-miner/DEPLOYMENT.md#running-fleet-mode-alongside-orbs-ams-observability-profile). + +## Alternative: ORB + laptop-mode AMS + +No fleet compose / override needed — the CLI already writes to the host directory the exporter mounts. + +```sh +# Terminal A — ORB + Grafana + AMS exporter +docker compose \ + -f docker-compose.yml \ + --profile observability \ + --profile ams-observability \ + up -d + +# Terminal B — AMS on the host (same LOOPOVER_MINER_CONFIG_DIR default) +loopover-miner doctor +loopover-miner run # or: loopover-miner loop +``` + +## Split commands (when you prefer separate projects) + +You can start ORB and AMS in two processes. Keep the **same** `LOOPOVER_MINER_CONFIG_DIR` in the environment for both, and pass the override whenever the fleet miner should feed Grafana: + +```sh +# ORB + observability + AMS exporter +docker compose -f docker-compose.yml \ + --profile observability --profile ams-observability up -d + +# AMS fleet (bridge override so ledgers land on the host path) +cp packages/loopover-miner/docker-compose.miner.override.yml.example \ + packages/loopover-miner/docker-compose.miner.override.yml +docker compose \ + -f packages/loopover-miner/docker-compose.miner.yml \ + -f packages/loopover-miner/docker-compose.miner.override.yml \ + up -d --build +``` + +Do **not** run fleet without the override and expect `ams-observability` panels to fill — named-volume paths are not the exporter's host bind. + +## Notes verified against current compose (not historical issue text) + +- `#5805` is **merged**: the opt-in override + DEPLOYMENT bridge section are the supported fleet↔exporter path. No further code change is required for path alignment. +- Root `docker-compose.yml` PROFILES header comment may still omit `ams-observability` / `backup` (tracked separately); the `ams-reporting-exporter` service **does** declare `profiles: ["ams-observability"]` and is what this quickstart activates. +- `packages/loopover-miner/docker-compose.miner.yml` currently tags the built image `gittensory-miner:latest` while the Dockerfile entrypoint is `loopover-miner` — cosmetic until that tag is renamed; compose `build` still produces a working local image. + +## Out of scope + +- Operational runbooks, alert tuning, and backup/restore — see AMS [`operations-runbook`](packages/loopover-miner/docs/operations-runbook.md) and ORB self-host docs. +- Scaling N concurrent fleet workers on one SQLite volume — unsafe; see DEPLOYMENT.md (separate compose projects or Kubernetes). diff --git a/README.md b/README.md index 601f4cbd6d..dbf9d2db60 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,7 @@ See [Tuning your reviews](https://gittensory.aethereal.dev/docs/tuning) for the | ------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Miners and contributors | [Quickstart](https://gittensory.aethereal.dev/docs/quickstart) | [MCP client setup](https://gittensory.aethereal.dev/docs/mcp-clients), [Miner workflow](https://gittensory.aethereal.dev/docs/miner-workflow), [Scoreability](https://gittensory.aethereal.dev/docs/scoreability) | | Maintainers | [GitHub App](https://gittensory.aethereal.dev/docs/github-app) | [Maintainer workflow](https://gittensory.aethereal.dev/docs/maintainer-workflow), [Self-host reviews](https://gittensory.aethereal.dev/docs/maintainer-self-hosting), [Privacy and security](https://gittensory.aethereal.dev/docs/privacy-security) | -| Repo owners and operators | [Beta onboarding](https://gittensory.aethereal.dev/docs/beta-onboarding) | [Upstream drift](https://gittensory.aethereal.dev/docs/upstream-drift), [Troubleshooting](https://gittensory.aethereal.dev/docs/troubleshooting), [Roadmap](https://gittensory.aethereal.dev/roadmap) | +| Repo owners and operators | [Beta onboarding](https://gittensory.aethereal.dev/docs/beta-onboarding) | [ORB + AMS together](ORB_AMS_QUICKSTART.md), [Upstream drift](https://gittensory.aethereal.dev/docs/upstream-drift), [Troubleshooting](https://gittensory.aethereal.dev/docs/troubleshooting), [Roadmap](https://gittensory.aethereal.dev/roadmap) | | Agent authors | [Agents](https://gittensory.aethereal.dev/agents) | [API browser](https://gittensory.aethereal.dev/api), [MCP client setup](https://gittensory.aethereal.dev/docs/mcp-clients) | ## Surfaces @@ -66,6 +66,7 @@ See [Tuning your reviews](https://gittensory.aethereal.dev/docs/tuning) for the | MCP package | [@loopover/mcp](https://www.npmjs.com/package/@loopover/mcp) | | Engine package | [`@loopover/engine`](packages/loopover-engine/README.md) — shared deterministic logic for the review stack and miner | | Miner package | [`@loopover/miner`](packages/loopover-miner/README.md) — local foundation CLI for the autonomous miner runtime | +| ORB + AMS (self-host) | [`ORB_AMS_QUICKSTART.md`](ORB_AMS_QUICKSTART.md) — run root `docker-compose.yml` and AMS fleet/laptop on one host | | API | [API browser](https://gittensory.aethereal.dev/api) and [OpenAPI JSON](https://gittensory-api.aethereal.dev/openapi.json) | | GitHub App | [Setup docs](https://gittensory.aethereal.dev/docs/github-app) — self-hosting is the only currently available path | | Browser extension | [Extension page](https://gittensory.aethereal.dev/extension) | @@ -97,6 +98,10 @@ loopover-mcp agent packet --login jsonbored --json loopover-mcp agent status --json ``` +## Running ORB + AMS together + +Self-host the review stack (ORB, root [`docker-compose.yml`](docker-compose.yml)) and the autonomous miner (AMS) on one machine — including the Grafana `ams-observability` bridge for fleet or laptop mode — with the copy-pasteable path in [`ORB_AMS_QUICKSTART.md`](ORB_AMS_QUICKSTART.md). AMS-only laptop/fleet details remain in [`packages/loopover-miner/DEPLOYMENT.md`](packages/loopover-miner/DEPLOYMENT.md). + ## Local Development ```sh diff --git a/packages/loopover-miner/DEPLOYMENT.md b/packages/loopover-miner/DEPLOYMENT.md index 1c3b2a93d2..f86c9afbba 100644 --- a/packages/loopover-miner/DEPLOYMENT.md +++ b/packages/loopover-miner/DEPLOYMENT.md @@ -137,9 +137,11 @@ docker compose -f docker-compose.miner.yml up -d --build ### Running fleet mode alongside ORB's `ams-observability` profile +For the full dual-product operator path (ORB + AMS env files, laptop vs fleet, and Grafana verification), start at the repo-root [`ORB_AMS_QUICKSTART.md`](../../ORB_AMS_QUICKSTART.md). The rest of this section is the AMS-side bridge detail. + Fleet mode keeps miner state in a named `miner-data` volume, but ORB's `ams-reporting-exporter` (root [`docker-compose.yml`](../../docker-compose.yml), `--profile ams-observability`) reads the miner's ledgers from a **host** directory (default `~/.config/loopover-miner`). A named volume's host path is a Docker-managed internal detail, so the two never line up on their own — the exporter reads an empty directory and the Grafana AMS datasources stay **silently empty**. -To bridge them, relocate the fleet miner's state onto a host directory with the opt-in override, then run both profiles together: +To bridge them, relocate the fleet miner's state onto a host directory with the opt-in override, then run both profiles together (add `--profile observability` when you also want Grafana up in the same invoke — see the quickstart): ```sh cp packages/loopover-miner/docker-compose.miner.override.yml.example \ diff --git a/packages/loopover-miner/README.md b/packages/loopover-miner/README.md index 8d2ee00d41..4eb5513ca0 100644 --- a/packages/loopover-miner/README.md +++ b/packages/loopover-miner/README.md @@ -163,7 +163,7 @@ See [`docs/cross-repo-discovery-phase1.md`](docs/cross-repo-discovery-phase1.md) See [`docs/discovery-plane-operator-guide.md`](docs/discovery-plane-operator-guide.md) for the optional hosted discovery-index plane (opt-in default OFF; contrasts with Orb's opt-out-only export — [#4309](https://github.com/JSONbored/gittensory/issues/4309)). -See [`DEPLOYMENT.md`](DEPLOYMENT.md) for laptop vs fleet deployment. +See [`DEPLOYMENT.md`](DEPLOYMENT.md) for laptop vs fleet deployment. To run AMS alongside the self-hosted ORB compose stack (including Grafana `ams-observability`), see the repo-root [`ORB_AMS_QUICKSTART.md`](../../ORB_AMS_QUICKSTART.md). See [`docs/operations-runbook.md`](docs/operations-runbook.md) for SQLite concurrency guarantees, corruption recovery, multi-process collision response, and post-upgrade ledger migration ([#4875](https://github.com/JSONbored/gittensory/issues/4875)).