Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions ORB_AMS_QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Running ORB + AMS together

Operator quickstart for self-hosting **ORB** (the review stack in the root [`docker-compose.yml`](docker-compose.yml)) and **AMS** (`@loopover/miner`, fleet or laptop) on **one host**, including the Grafana `ams-observability` bridge.

Each product works alone. This page is the single path that documents how their compose files, volumes, and `LOOPOVER_MINER_CONFIG_DIR` interact. AMS-only deploy details stay in [`packages/loopover-miner/DEPLOYMENT.md`](packages/loopover-miner/DEPLOYMENT.md); AMS Grafana wiring stays in [`packages/loopover-miner/docs/observability.md`](packages/loopover-miner/docs/observability.md).

## What you get

| Piece | Compose file | Role |
| ----- | ------------ | ---- |
| ORB | root `docker-compose.yml` | Self-hosted LoopOver API / review agent (and optional Prometheus + Grafana) |
| AMS fleet worker | `packages/loopover-miner/docker-compose.miner.yml` | Long-lived miner container (`command: ["run"]`) |
| AMS → Grafana bridge | same root compose, `--profile ams-observability` | `ams-reporting-exporter` redacts miner ledgers into Grafana's reporting volume |

Grafana itself lives under `--profile observability`. The AMS exporter is a **separate** profile (`ams-observability`) so engine-only deployments do not start it. For panels to appear you need **both** profiles (plus a miner writing ledgers the exporter can read).

## Prerequisites

1. Docker Compose v2 on the host.
2. ORB secrets: copy [`.env.selfhost.example`](.env.selfhost.example) (or [`.env.example`](.env.example)) → `.env` at the repo root and fill GitHub App credentials. Never commit real values.
3. AMS secrets (fleet path): from `packages/loopover-miner/`:

```sh
cp .loopover-miner.env.example .loopover-miner.env
# fill GITHUB_TOKEN (+ optional coding-agent provider keys)
```

4. Optional — custom shared state directory (defaults are fine):

```sh
# In the repo-root `.env` (and exported in the shell if you prefer):
LOOPOVER_MINER_CONFIG_DIR=~/.config/loopover-miner
```

## Env vars that must agree

| Variable | ORB side | AMS side | Notes |
| -------- | -------- | -------- | ----- |
| `LOOPOVER_MINER_CONFIG_DIR` | Host path mounted **read-only** on `ams-reporting-exporter` at `/ams-ledgers` (default `~/.config/loopover-miner`) | **Laptop:** where the CLI writes SQLite. **Fleet + override:** host path bind-mounted at the container's `/data/miner` | Set once; both sides use the same default when unset. |
| AMS credentials | Not shared with ORB | `.loopover-miner.env` (`GITHUB_TOKEN`, …) | ORB uses the root `.env` GitHub App material; do not mix the two secret files. |

### Laptop mode vs fleet mode state

| AMS mode | Where ledgers live by default | Extra step for `ams-observability` |
| -------- | ----------------------------- | ---------------------------------- |
| Laptop (`loopover-miner` on the host) | `~/.config/loopover-miner` | None — matches the exporter bind mount |
| Fleet (`docker-compose.miner.yml` alone) | Named Docker volume `miner-data` → `/data/miner` | **Required:** copy the opt-in override so fleet state is a **host** directory the exporter can read (see below). Without it, Grafana AMS datasources stay **silently empty**. |

The fleet bridge is the `#5805` override in [`packages/loopover-miner/docker-compose.miner.override.yml.example`](packages/loopover-miner/docker-compose.miner.override.yml.example). It replaces the named-volume mount for `/data/miner` with `${LOOPOVER_MINER_CONFIG_DIR:-~/.config/loopover-miner}:/data/miner` — the same expression `ams-reporting-exporter` already uses.

## Worked example: ORB + AMS fleet + Grafana AMS panels

Run from the **monorepo root**. This is the verified two-product invocation (ORB profiles + AMS fleet compose + fleet↔exporter bridge).

```sh
# 1) One-time: opt-in host bind so fleet state matches the exporter (gitignored after copy)
cp packages/loopover-miner/docker-compose.miner.override.yml.example \
packages/loopover-miner/docker-compose.miner.override.yml

# 2) Stand up ORB (API + observability stack) and AMS exporter + fleet miner together
docker compose \
-f docker-compose.yml \
-f packages/loopover-miner/docker-compose.miner.yml \
-f packages/loopover-miner/docker-compose.miner.override.yml \
--profile observability \
--profile ams-observability \
up -d --build
```

### Expected output (shape)

`docker compose … up -d --build` should finish with services including at least:

- ORB core (e.g. `loopover`, Redis / SQLite stack depending on your base profiles)
- `grafana`, `prometheus`, … from `--profile observability`
- `ams-reporting-exporter` from `--profile ams-observability`
- `miner` from `docker-compose.miner.yml`

Then:

```sh
docker compose \
-f docker-compose.yml \
-f packages/loopover-miner/docker-compose.miner.yml \
-f packages/loopover-miner/docker-compose.miner.override.yml \
--profile observability \
--profile ams-observability \
ps
```

`ams-reporting-exporter` and `miner` should be `running`. After the miner has written `attempt-log.sqlite3` / `prediction-ledger.sqlite3` under `LOOPOVER_MINER_CONFIG_DIR`, the exporter's healthcheck expects a non-empty redacted `/reporting/ams-attempt-log.sqlite` (default interval 30s).

### Confirm Grafana AMS panels (fleet mode)

1. Open Grafana (default published port from the observability profile — see your `.env` / compose port mapping).
2. **Connections → Data sources** should list **AMS Attempt Log** and **AMS Prediction Ledger** (provisioned from [`grafana/provisioning/datasources/ams-ledgers.yml`](grafana/provisioning/datasources/ams-ledgers.yml)).
3. Explore either datasource after at least one export cycle. Empty tables before the miner has logged attempts is normal; **empty forever** usually means the exporter and miner still disagree on the host path (missing override, or a custom `LOOPOVER_MINER_CONFIG_DIR` set on only one side).

Deep dive on what is exported vs kept private: [`packages/loopover-miner/docs/observability.md`](packages/loopover-miner/docs/observability.md). Bridge rationale and AMS-only wording: [DEPLOYMENT.md — Running fleet mode alongside ORB's `ams-observability` profile](packages/loopover-miner/DEPLOYMENT.md#running-fleet-mode-alongside-orbs-ams-observability-profile).

## Alternative: ORB + laptop-mode AMS

No fleet compose / override needed — the CLI already writes to the host directory the exporter mounts.

```sh
# Terminal A — ORB + Grafana + AMS exporter
docker compose \
-f docker-compose.yml \
--profile observability \
--profile ams-observability \
up -d

# Terminal B — AMS on the host (same LOOPOVER_MINER_CONFIG_DIR default)
loopover-miner doctor
loopover-miner run # or: loopover-miner loop
```

## Split commands (when you prefer separate projects)

You can start ORB and AMS in two processes. Keep the **same** `LOOPOVER_MINER_CONFIG_DIR` in the environment for both, and pass the override whenever the fleet miner should feed Grafana:

```sh
# ORB + observability + AMS exporter
docker compose -f docker-compose.yml \
--profile observability --profile ams-observability up -d

# AMS fleet (bridge override so ledgers land on the host path)
cp packages/loopover-miner/docker-compose.miner.override.yml.example \
packages/loopover-miner/docker-compose.miner.override.yml
docker compose \
-f packages/loopover-miner/docker-compose.miner.yml \
-f packages/loopover-miner/docker-compose.miner.override.yml \
up -d --build
```

Do **not** run fleet without the override and expect `ams-observability` panels to fill — named-volume paths are not the exporter's host bind.

## Notes verified against current compose (not historical issue text)

- `#5805` is **merged**: the opt-in override + DEPLOYMENT bridge section are the supported fleet↔exporter path. No further code change is required for path alignment.
- Root `docker-compose.yml` PROFILES header comment may still omit `ams-observability` / `backup` (tracked separately); the `ams-reporting-exporter` service **does** declare `profiles: ["ams-observability"]` and is what this quickstart activates.
- `packages/loopover-miner/docker-compose.miner.yml` currently tags the built image `gittensory-miner:latest` while the Dockerfile entrypoint is `loopover-miner` — cosmetic until that tag is renamed; compose `build` still produces a working local image.

## Out of scope

- Operational runbooks, alert tuning, and backup/restore — see AMS [`operations-runbook`](packages/loopover-miner/docs/operations-runbook.md) and ORB self-host docs.
- Scaling N concurrent fleet workers on one SQLite volume — unsafe; see DEPLOYMENT.md (separate compose projects or Kubernetes).
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ See [Tuning your reviews](https://gittensory.aethereal.dev/docs/tuning) for the
| ------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Miners and contributors | [Quickstart](https://gittensory.aethereal.dev/docs/quickstart) | [MCP client setup](https://gittensory.aethereal.dev/docs/mcp-clients), [Miner workflow](https://gittensory.aethereal.dev/docs/miner-workflow), [Scoreability](https://gittensory.aethereal.dev/docs/scoreability) |
| Maintainers | [GitHub App](https://gittensory.aethereal.dev/docs/github-app) | [Maintainer workflow](https://gittensory.aethereal.dev/docs/maintainer-workflow), [Self-host reviews](https://gittensory.aethereal.dev/docs/maintainer-self-hosting), [Privacy and security](https://gittensory.aethereal.dev/docs/privacy-security) |
| Repo owners and operators | [Beta onboarding](https://gittensory.aethereal.dev/docs/beta-onboarding) | [Upstream drift](https://gittensory.aethereal.dev/docs/upstream-drift), [Troubleshooting](https://gittensory.aethereal.dev/docs/troubleshooting), [Roadmap](https://gittensory.aethereal.dev/roadmap) |
| Repo owners and operators | [Beta onboarding](https://gittensory.aethereal.dev/docs/beta-onboarding) | [ORB + AMS together](ORB_AMS_QUICKSTART.md), [Upstream drift](https://gittensory.aethereal.dev/docs/upstream-drift), [Troubleshooting](https://gittensory.aethereal.dev/docs/troubleshooting), [Roadmap](https://gittensory.aethereal.dev/roadmap) |
| Agent authors | [Agents](https://gittensory.aethereal.dev/agents) | [API browser](https://gittensory.aethereal.dev/api), [MCP client setup](https://gittensory.aethereal.dev/docs/mcp-clients) |

## Surfaces
Expand All @@ -66,6 +66,7 @@ See [Tuning your reviews](https://gittensory.aethereal.dev/docs/tuning) for the
| MCP package | [@loopover/mcp](https://www.npmjs.com/package/@loopover/mcp) |
| Engine package | [`@loopover/engine`](packages/loopover-engine/README.md) — shared deterministic logic for the review stack and miner |
| Miner package | [`@loopover/miner`](packages/loopover-miner/README.md) — local foundation CLI for the autonomous miner runtime |
| ORB + AMS (self-host) | [`ORB_AMS_QUICKSTART.md`](ORB_AMS_QUICKSTART.md) — run root `docker-compose.yml` and AMS fleet/laptop on one host |
| API | [API browser](https://gittensory.aethereal.dev/api) and [OpenAPI JSON](https://gittensory-api.aethereal.dev/openapi.json) |
| GitHub App | [Setup docs](https://gittensory.aethereal.dev/docs/github-app) — self-hosting is the only currently available path |
| Browser extension | [Extension page](https://gittensory.aethereal.dev/extension) |
Expand Down Expand Up @@ -97,6 +98,10 @@ loopover-mcp agent packet --login jsonbored --json
loopover-mcp agent status <run-id> --json
```

## Running ORB + AMS together

Self-host the review stack (ORB, root [`docker-compose.yml`](docker-compose.yml)) and the autonomous miner (AMS) on one machine — including the Grafana `ams-observability` bridge for fleet or laptop mode — with the copy-pasteable path in [`ORB_AMS_QUICKSTART.md`](ORB_AMS_QUICKSTART.md). AMS-only laptop/fleet details remain in [`packages/loopover-miner/DEPLOYMENT.md`](packages/loopover-miner/DEPLOYMENT.md).

## Local Development

```sh
Expand Down
4 changes: 3 additions & 1 deletion packages/loopover-miner/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,9 +137,11 @@ docker compose -f docker-compose.miner.yml up -d --build

### Running fleet mode alongside ORB's `ams-observability` profile

For the full dual-product operator path (ORB + AMS env files, laptop vs fleet, and Grafana verification), start at the repo-root [`ORB_AMS_QUICKSTART.md`](../../ORB_AMS_QUICKSTART.md). The rest of this section is the AMS-side bridge detail.

Fleet mode keeps miner state in a named `miner-data` volume, but ORB's `ams-reporting-exporter` (root [`docker-compose.yml`](../../docker-compose.yml), `--profile ams-observability`) reads the miner's ledgers from a **host** directory (default `~/.config/loopover-miner`). A named volume's host path is a Docker-managed internal detail, so the two never line up on their own — the exporter reads an empty directory and the Grafana AMS datasources stay **silently empty**.

To bridge them, relocate the fleet miner's state onto a host directory with the opt-in override, then run both profiles together:
To bridge them, relocate the fleet miner's state onto a host directory with the opt-in override, then run both profiles together (add `--profile observability` when you also want Grafana up in the same invoke — see the quickstart):

```sh
cp packages/loopover-miner/docker-compose.miner.override.yml.example \
Expand Down
2 changes: 1 addition & 1 deletion packages/loopover-miner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ See [`docs/cross-repo-discovery-phase1.md`](docs/cross-repo-discovery-phase1.md)

See [`docs/discovery-plane-operator-guide.md`](docs/discovery-plane-operator-guide.md) for the optional hosted discovery-index plane (opt-in default OFF; contrasts with Orb's opt-out-only export — [#4309](https://github.com/JSONbored/gittensory/issues/4309)).

See [`DEPLOYMENT.md`](DEPLOYMENT.md) for laptop vs fleet deployment.
See [`DEPLOYMENT.md`](DEPLOYMENT.md) for laptop vs fleet deployment. To run AMS alongside the self-hosted ORB compose stack (including Grafana `ams-observability`), see the repo-root [`ORB_AMS_QUICKSTART.md`](../../ORB_AMS_QUICKSTART.md).

See [`docs/operations-runbook.md`](docs/operations-runbook.md) for SQLite concurrency guarantees, corruption recovery, multi-process collision response, and post-upgrade ledger migration ([#4875](https://github.com/JSONbored/gittensory/issues/4875)).

Expand Down