feat(selfhost): observability stack — logs, alerts, dashboards, reliability hardening

[JSONbored]

Summary

Makes the self-host Docker stack production-observable and resilient out of the box. One command — docker compose --profile observability up -d — now brings up metrics, logs, alerting, and dashboards, and the core stack is hardened for running real PR reviews.

Logs (new): Loki (TSDB schema v13, filesystem store, 14-day compactor retention) + Promtail (discovers the compose project's containers over a read-only Docker socket; parses the app's JSON log lines into service/level labels with event as structured metadata) + a provisioned Grafana Loki datasource. New dashboard rows: live service logs, errors-only, HTTP status/latency, and active alerts.

Alerts (new): 9 Prometheus rules grounded only in real exported metrics — target down, job-failure ratio, dead-letter growth + backlog, queue backlog, Qdrant error rate, Orb export error rate, HTTP 5xx ratio, p95 latency SLO — wired to Alertmanager, which ships silent (null receiver) with commented Slack/email/webhook examples so notifications work the moment an operator fills one in.

Metrics: gittensory_http_requests_total gains a status class label, and a new gittensory_http_request_duration_seconds histogram is added. metrics.ts gains histogram support (observe() + Prometheus _bucket/_sum/_count rendering).

Reliability hardening:

• Healthchecks move from /health (a liveness stub that's 200 even with the DB down) to /ready (DB answers + migrations applied), with a 60s start_period for the Postgres cold start.
• Resilient retry around the Qdrant init (a single no-retry fetch that crash-loops on a slow cold start), plus a compose depends_on gate on a real Qdrant /readyz healthcheck.
• Redis hardened to a bounded cache-only role (256mb, allkeys-lru, RDB+AOF off, volume dropped — it's ephemeral).
• Qdrant ports bound to loopback (the app uses the internal network; the local container is unauthenticated) — matches the in-network-only posture of Prometheus/Loki/Alertmanager.
• Orb export failures are now logged (were silently swallowed).

Docs: PUBLIC_API_ORIGIN (required by the setup wizards), Qdrant auth/dimension, and the observability log stack documented in .env.example.

Scope

• Self-host (src/server.ts, src/selfhost/metrics.ts) + Docker stack (compose, Dockerfile, prometheus/loki/promtail/alertmanager/grafana configs) + .env.example. No changes to the Cloudflare Worker path.
• In wantedPaths; no blockedPaths; no CHANGELOG.md.
• New capabilities are opt-in behind --profile observability / --profile qdrant; the default stack is unchanged except the healthcheck endpoint.

Validation

Run locally (all green):

• npm run test:ci — exit 0
• npm audit --audit-level=moderate — 0 vulnerabilities
• npm run typecheck, npx vitest run test/unit/selfhost-metrics.test.ts (histogram tests, 100% branch coverage on metrics.ts)
• promtool check config / check rules — valid, 9 rules; amtool check-config — valid
• docker compose config renders for the default stack and every profile combination

Validated live in Docker (full postgres + redis + qdrant + observability stack):

• Stack comes up healthy; /ready 200 on Postgres (60 migrations); pg-adapter exercised live
• /metrics shows the new status classes (2xx/4xx) + latency histogram; p95 resolves (0.067s)
• Logs flow into Loki ({service="gittensory"}, level filter); Promtail discovers all containers
• Prometheus scrapes the app, loads 9 rules, registers Alertmanager
• Alert fires end-to-end: stopping the app raised GittensoryTargetDown (critical) → Prometheus → Alertmanager, and it auto-resolved on recovery
• Redis hardening confirmed via redis-cli config get; rate limiter writes keys under real traffic
• Qdrant healthcheck gates app startup; collection auto-created; ports bound to 127.0.0.1

Safety

• No secrets/wallets/hotkeys/trust-scores/reward values in code, configs, or docs. Alertmanager receivers are commented placeholders. The local Redis/Qdrant are internal-only/loopback-only.
• Promtail's Docker socket mount is read-only (:ro).
• No public-comment / private-boundary surfaces touched.
• No UI (apps/gittensory-ui) changes — the Grafana dashboard is observability config (JSON), not the product UI, so no UI Evidence table applies.

Extends the self-host observability work (#980 / #1206). No behavior change to the hosted Worker.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.03%. Comparing base (654e7ca) to head (b726d1f).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1239   +/-   ##
=======================================
  Coverage   95.03%   95.03%           
=======================================
  Files         177      177           
  Lines       19894    19913   +19     
  Branches     7170     7172    +2     
=======================================
+ Hits        18906    18925   +19     
  Misses        395      395           
  Partials      593      593           

Files with missing lines	Coverage Δ
src/selfhost/metrics.ts	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[superagent-security]

Superagent found 1 security concern(s).

[superagent-security]

P2: Promtail Docker socket mount mischaracterized as read-only safe

The :ro Docker socket mount does not block API writes as claimed.

Correct the comment and consider a Docker socket proxy to restrict API access.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name="docker-compose.yml">
<violation number="1" location="docker-compose.yml:296">
<priority>P2</priority>
<title>Promtail Docker socket mount mischaracterized as read-only safe</title>
<evidence>The comment on line 294 claims the :ro mount &quot;blocks any writes to the daemon&quot;, but a UNIX socket bind-mount with :ro only prevents modification of the socket inode. It does NOT restrict Docker API operations. A compromised Promtail container can still use the socket to enumerate containers, inspect configs (including env vars with secrets), read all container logs, and potentially create a privileged container to escape to the host. The same file already acknowledges on line 341 that mounting the Docker socket &quot;grants container-escape risk&quot;, contradicting the false reassurance here.</evidence>
<recommendation>Add an accurate security warning: explain that :ro on a socket does not restrict Docker API calls, that the socket grants effective host root access, and consider adding a Docker socket proxy (such as tecnativa/docker-socket-proxy) that exposes only the read-only subset of the API Promtail actually needs.</recommendation>
</violation>
</file>

[superagent-security]

P1: Docker socket proxy permits inspect API exposing all container secrets

CONTAINERS: "1" allows GET /containers/{id}/json, which returns full container configs including Config.Env with every secret.

Isolate the proxy on a dedicated network only Promtail can reach, or use Docker secrets to keep credentials out of environment variables.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name="docker-compose.yml">
<violation number="1" location="docker-compose.yml:299">
<priority>P1</priority>
<title>Docker socket proxy permits inspect API exposing all container secrets</title>
<evidence>The docker-proxy service mounts /var/run/docker.sock:ro and is configured with CONTAINERS: "1", which permits all GET /containers/* endpoints. This includes GET /containers/{id}/json, which returns the full container configuration object containing Config.Env — the complete list of environment variables for every container. Since the default Docker Compose network allows all services to communicate, any compromised container in the project can connect to docker-proxy:2375 and exfiltrate secrets such as POSTGRES_PASSWORD, GRAFANA_ADMIN_PASSWORD, database URLs, and API keys from every other container.</evidence>
<recommendation>Place the docker-proxy service on a dedicated internal Docker network that is only shared with Promtail, so other containers cannot reach the proxy. Alternatively, migrate sensitive values from environment variables to Docker secrets or mounted secret files so they do not appear in Config.Env via the inspect endpoint.</recommendation>
</violation>
</file>

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.