Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/selfhost.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,6 @@ jobs:
if [ "$ok" != "1" ]; then echo "::error::container did not become healthy"; docker logs gt; exit 1; fi
curl -sf http://127.0.0.1:8787/health | grep -q '"status":"ok"'
curl -sf http://127.0.0.1:8787/ready | grep -q '"ok":true'
curl -sf http://127.0.0.1:8787/metrics | grep -q 'gittensory_uptime_seconds'
curl -sf http://127.0.0.1:8787/metrics | grep -q 'loopover_uptime_seconds'
docker logs gt 2>&1 | grep -q 'selfhost_migrations_applied'
echo "self-host smoke test passed"
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ GITHUB_METADATA_CACHE_TTL_SECONDS=600`}
above so operators can keep repeated policy reads hot without broadening stale cache risk.
Live CI status, check-run, check-suite, pull/issue subresources, pull mergeability, token
minting, rate-limit, and collaborator-permission endpoints are never served from this cache.
Prometheus exports <code>gittensory_github_response_cache_total</code>, and the bundled
Prometheus exports <code>loopover_github_response_cache_total</code>, and the bundled
self-host Grafana dashboard includes the hit/miss/coalesced/error breakdown.
</Callout>

Expand Down
14 changes: 7 additions & 7 deletions apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,12 @@ docker compose --profile postgres --profile observability --profile backup up -d
Check sync health with <code>chronyc sources</code> (or <code>ntpq -p</code> on an{" "}
<code>ntpd</code> host) — every configured source should show a nonzero <code>Reach</code>{" "}
value; <code>Reach: 0</code> means that source has never successfully synced. The{" "}
<code>gittensory_clock_skew_seconds</code> gauge on the <strong>Clock Sync (NTP)</strong>{" "}
row of the main Grafana dashboard tracks the live drift between this process and GitHub's
server time, sampled from the <code>Date</code> header of the GitHub App's own
installation-token mint calls — no extra network probe required. The bundled Prometheus
rules alert at 60s (warning) and 120s (critical) drift, both well under the margin that
actually breaks JWT auth.
<code>loopover_clock_skew_seconds</code> gauge on the <strong>Clock Sync (NTP)</strong> row
of the main Grafana dashboard tracks the live drift between this process and GitHub's server
time, sampled from the <code>Date</code> header of the GitHub App's own installation-token
mint calls — no extra network probe required. The bundled Prometheus rules alert at 60s
(warning) and 120s (critical) drift, both well under the margin that actually breaks JWT
auth.
</p>

<h2>Alerting — required for a 24/7 deployment</h2>
Expand Down Expand Up @@ -650,7 +650,7 @@ SENTRY_RELEASE=gittensory-selfhost@2026.07.05
</td>
<td className="py-2 align-top text-muted-foreground">
Check Grafana dead-job panel, <code>/metrics</code>{" "}
<code>gittensory_jobs_dead_total</code>, queue logs; replay from DLQ dashboard only
<code>loopover_jobs_dead_total</code>, queue logs; replay from DLQ dashboard only
after fixing root cause
</td>
</tr>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ function SelfHostingTroubleshooting() {
<CodeBlock
lang="bash"
code={`docker compose ps
docker compose logs --tail=200 gittensory
docker compose logs --tail=200 loopover
curl http://localhost:8787/ready
curl http://localhost:8787/metrics`}
/>
Expand Down Expand Up @@ -122,18 +122,17 @@ rees_analyzer_config_invalid`}
</p>
<CodeBlock
lang="bash"
code={`curl http://localhost:8787/metrics | grep gittensory_queue
docker compose logs gittensory | grep selfhost_job_dead`}
code={`curl http://localhost:8787/metrics | grep loopover_queue
docker compose logs loopover | grep selfhost_job_dead`}
/>

<h2>GitHub rate-limit responses or admission deferrals</h2>
<p>
Two independent signals cover this:{" "}
<code>gittensory_github_rest_rate_limit_responses_total</code> counts actual 403/429
responses from GitHub, and the{" "}
<code>gittensory_jobs_rate_limit_admission_deferred_total</code> /{" "}
<code>gittensory_jobs_rate_limit_budget_deferred_total</code> /{" "}
<code>gittensory_jobs_rate_limited_by_type_total</code> counters track jobs the queue itself
<code>loopover_github_rest_rate_limit_responses_total</code> counts actual 403/429 responses
from GitHub, and the <code>loopover_jobs_rate_limit_admission_deferred_total</code> /{" "}
<code>loopover_jobs_rate_limit_budget_deferred_total</code> /{" "}
<code>loopover_jobs_rate_limited_by_type_total</code> counters track jobs the queue itself
held back <em>before</em> making a request, to avoid tripping a limit. All three job-side
counters carry the same three labels — <code>kind</code> (<code>webhook</code> or{" "}
<code>background</code>), <code>key_scope</code> (<code>installation</code>,{" "}
Expand All @@ -145,20 +144,20 @@ docker compose logs gittensory | grep selfhost_job_dead`}
A short burst of deferrals is expected and self-resolving: the queue is deliberately trading
a few seconds of delay to avoid a real 429. Treat it as a real problem only once it&apos;s
<strong> sustained</strong> — which is exactly what{" "}
<code>GittensoryGitHubRateLimitResponses</code> (real 403/429s observed) and{" "}
<code>GittensoryQueueRateLimitDeferralsHigh</code> (a sustained deferral rate, not a blip)
are tuned to alert on, rather than firing on every brief admission hold.
<code>LoopoverGitHubRateLimitResponses</code> (real 403/429s observed) and{" "}
<code>LoopoverQueueRateLimitDeferralsHigh</code> (a sustained deferral rate, not a blip) are
tuned to alert on, rather than firing on every brief admission hold.
</p>
<CodeBlock
lang="promql"
code={`# Deferrals broken down by token pool and job type over the last 10m
sum by (key_scope, job_type) (rate(gittensory_jobs_rate_limit_admission_deferred_total[10m]))
sum by (key_scope, job_type) (rate(loopover_jobs_rate_limit_admission_deferred_total[10m]))

# Is one key_scope (e.g. a single installation token) the bottleneck?
topk(5, sum by (key_scope) (rate(gittensory_jobs_rate_limit_budget_deferred_total[10m])))
topk(5, sum by (key_scope) (rate(loopover_jobs_rate_limit_budget_deferred_total[10m])))

# Real rate-limit responses from GitHub itself (not just internal deferrals)
sum(rate(gittensory_github_rest_rate_limit_responses_total[10m]))`}
sum(rate(loopover_github_rest_rate_limit_responses_total[10m]))`}
/>
<p>
If a single <code>key_scope=installation</code> pool is consistently the bottleneck, the fix
Expand All @@ -168,33 +167,33 @@ sum(rate(gittensory_github_rest_rate_limit_responses_total[10m]))`}

<h2>Low GitHub response-cache hit rate</h2>
<p>
<code>gittensory_github_response_cache_total</code> (REST) and{" "}
<code>gittensory_github_graphql_cache_total</code> (GraphQL) both carry a{" "}
<code>result</code> label — <code>hit</code>, <code>miss</code>, <code>set</code>,{" "}
<code>coalesced</code>, <code>bypassed</code>, or <code>error</code> — and a{" "}
<code>class</code> label identifying the endpoint family. A healthy cache should show most
traffic as <code>hit</code> for endpoints that are read repeatedly in one review/maintenance
pass (PR reads, check-run lookups); a low hit rate on those specific classes, not the
overall average, is the useful signal.
<code>loopover_github_response_cache_total</code> (REST) and{" "}
<code>loopover_github_graphql_cache_total</code> (GraphQL) both carry a <code>result</code>{" "}
label — <code>hit</code>, <code>miss</code>, <code>set</code>, <code>coalesced</code>,{" "}
<code>bypassed</code>, or <code>error</code> — and a <code>class</code> label identifying
the endpoint family. A healthy cache should show most traffic as <code>hit</code> for
endpoints that are read repeatedly in one review/maintenance pass (PR reads, check-run
lookups); a low hit rate on those specific classes, not the overall average, is the useful
signal.
</p>
<CodeBlock
lang="promql"
code={`# REST hit rate by endpoint class over the last 15m
sum by (class) (rate(gittensory_github_response_cache_total{result="hit"}[15m]))
sum by (class) (rate(loopover_github_response_cache_total{result="hit"}[15m]))
/
sum by (class) (rate(gittensory_github_response_cache_total[15m]))
sum by (class) (rate(loopover_github_response_cache_total[15m]))

# GraphQL hit rate — same shape, separate metric
sum by (class) (rate(gittensory_github_graphql_cache_total{result="hit"}[15m]))
sum by (class) (rate(loopover_github_graphql_cache_total{result="hit"}[15m]))
/
sum by (class) (rate(gittensory_github_graphql_cache_total[15m]))`}
sum by (class) (rate(loopover_github_graphql_cache_total[15m]))`}
/>

<h2>Qdrant / vector-store errors</h2>
<p>
<code>gittensory_qdrant_errors_total</code> carries an <code>op</code> label (
<code>loopover_qdrant_errors_total</code> carries an <code>op</code> label (
<code>upsert</code>, <code>query</code>, or <code>delete</code>) so you can tell whether
indexing or retrieval is failing. <code>GittensoryQdrantErrorRateHigh</code> fires on a
indexing or retrieval is failing. <code>LoopoverQdrantErrorRateHigh</code> fires on a
sustained error ratio, not an isolated blip.
</p>
<ul>
Expand Down Expand Up @@ -226,14 +225,14 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}

<h2>Orb export or relay problems</h2>
<p>
For brokered self-host deployments, <code>gittensory_orb_events_exported_total</code> and{" "}
<code>gittensory_orb_export_errors_total</code> track the hourly outcome-export loop;{" "}
<code>GittensoryOrbExportErrorRateHigh</code> fires on a sustained error ratio there. The
For brokered self-host deployments, <code>loopover_orb_events_exported_total</code> and{" "}
<code>loopover_orb_export_errors_total</code> track the hourly outcome-export loop;{" "}
<code>LoopoverOrbExportErrorRateHigh</code> fires on a sustained error ratio there. The
pull-mode relay loop (for installations receiving events outbound from Orb) reports through{" "}
<code>gittensory_orb_relay_drains_total</code> (<code>result=events</code> when it drained
something, <code>result=empty</code> otherwise) and{" "}
<code>gittensory_orb_webhook_total</code> (<code>event</code> + <code>result</code> labels)
for what happened to each relayed event once enqueued locally.
<code>loopover_orb_relay_drains_total</code> (<code>result=events</code> when it drained
something, <code>result=empty</code> otherwise) and <code>loopover_orb_webhook_total</code>{" "}
(<code>event</code> + <code>result</code> labels) for what happened to each relayed event
once enqueued locally.
</p>
<p>
If exports are failing but the relay itself looks healthy, the export loop&apos;s Sentry
Expand All @@ -257,7 +256,7 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}
</li>
<li>
<strong>Pull mode</strong> — logged at <code>warn</code> and non-fatal: the drain loop (
<code>gittensory_orb_relay_drains_total</code>) keeps retrying on its own schedule
<code>loopover_orb_relay_drains_total</code>) keeps retrying on its own schedule
regardless, so a transient failure here recovers on its own once the broker is reachable
again. A registration failure that never clears across many retries still points at{" "}
<code>ORB_ENROLLMENT_SECRET</code> being wrong, revoked, or not yet provisioned
Expand All @@ -274,9 +273,9 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}
<p>
Each AI provider (self-host <code>AI_PROVIDER</code> entries) has its own circuit breaker:
after 3 consecutive failures it stops attempting real calls to that provider for 60 seconds,
recorded as <code>gittensory_ai_provider_circuit_open_total{'{provider="..."}'}</code>{" "}
recorded as <code>loopover_ai_provider_circuit_open_total{'{provider="..."}'}</code>{" "}
(skipped calls) alongside{" "}
<code>gittensory_ai_provider_failures_total{'{provider="..."}'}</code> (real failures). It
<code>loopover_ai_provider_failures_total{'{provider="..."}'}</code> (real failures). It
self-heals automatically — there is no manual reset — but it will reopen immediately if the
underlying problem is still there.
</p>
Expand All @@ -293,9 +292,9 @@ curl -X DELETE "$QDRANT_URL/collections/gittensory"`}
unreachable from the container.
</li>
<li>
<code>GittensoryAiProviderCircuitOpen</code> fires on any circuit-open event in a
15-minute window — a single trip during a real but brief outage is expected; a rule that
keeps firing across multiple windows points at the persistent case above.
<code>LoopoverAiProviderCircuitOpen</code> fires on any circuit-open event in a 15-minute
window — a single trip during a real but brief outage is expected; a rule that keeps
firing across multiple windows points at the persistent case above.
</li>
</ul>

Expand Down Expand Up @@ -326,7 +325,7 @@ npm run test:smoke:observability:metrics`}
</li>
<li>
If the app's own <code>/metrics</code> check fails, that is unrelated to the OTEL
collector — check the app container directly (<code>docker compose logs gittensory</code>
collector — check the app container directly (<code>docker compose logs loopover</code>
).
</li>
<li>
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
# (chrony/ntpd), not just one. A single dead NTP source silently drifts the clock with no local
# symptom until GitHub App JWT auth starts failing ("Bad credentials") once the drift exceeds the
# JWT's 60s skew tolerance. See docs/self-hosting/operations → "Host clock sync (NTP)" and the
# gittensory_clock_skew_seconds Grafana panel/alert (--profile observability) for live drift.
# loopover_clock_skew_seconds Grafana panel/alert (--profile observability) for live drift.

# Bounded container logging (#audit-rate-headroom): every service below defaults to Docker's
# json-file driver, which has NO size cap on its own -- a long-running 24/7 stack can fill the
Expand Down Expand Up @@ -1160,7 +1160,7 @@ services:
- -c
- "apk add --no-cache busybox-extras && sh /scripts/backup-metrics.sh"
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9101/metrics | grep -q '^gittensory_backup_latest_timestamp_seconds'"]
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9101/metrics | grep -q '^loopover_backup_latest_timestamp_seconds'"]
interval: 30s
retries: 5

Expand Down
Loading
Loading