Skip to content

fix(selfhost): .env.example QUEUE_CONCURRENCY guidance references nonexistent gittensory_queue_* metrics #5936

Description

@JSONbored

Context

.env.example's QUEUE_CONCURRENCY documentation block tells an operator raising concurrency to watch two
specific Prometheus series to see if the change helped:

# QUEUE_CONCURRENCY=4                        # ...
#                                            #   Watch gittensory_queue_live_pending
#                                            #   / gittensory_queue_oldest_live_pending_age_seconds after raising it;
#                                            #   if those stay high, the bottleneck is elsewhere...

Neither gittensory_queue_live_pending nor gittensory_queue_oldest_live_pending_age_seconds exists as a
metric name anywhere in the codebase (confirmed via a repo-wide grep for the gittensory_ metric prefix —
this is the only place it still appears). The real, currently-registered metrics use the loopover_ prefix
throughout src/selfhost/metrics.ts and prometheus/rules/alerts.yml:

  • loopover_queue_live_pending (used by prometheus/rules/alerts.yml's LoopoverLiveQueueStuckBehindMaintenance
    / LoopoverLiveQueueStarved alert expressions)
  • loopover_queue_oldest_live_pending_age_seconds (registered in src/selfhost/metrics.ts)

Practical effect: an operator who raises QUEUE_CONCURRENCY, follows this comment's advice, and searches
Grafana/Prometheus for gittensory_queue_live_pending gets an empty series with no explanation — the same
"comment references a metric that doesn't exist" failure mode as #5816 (filed earlier today for a stale
runbook annotation), just in a different file.

Requirements

  • Update both metric names in the QUEUE_CONCURRENCY comment block from the gittensory_ prefix to the real
    loopover_ prefix: loopover_queue_live_pending and loopover_queue_oldest_live_pending_age_seconds.
  • No other change to this section's guidance — the advice itself (watch these two series after raising
    concurrency) is accurate; only the metric names are stale.

Deliverables

  • .env.example: QUEUE_CONCURRENCY comment block updated to reference loopover_queue_live_pending /
    loopover_queue_oldest_live_pending_age_seconds.
  • A regression test (e.g. under test/unit/) that greps .env.example for the gittensory_ metric
    prefix and fails if any match is found, preventing this class of drift from recurring anywhere in the file.

Test Coverage Requirements

.env.example is a config/docs file outside src/** and Codecov's coverage.include, so patch coverage
does not gate this directly. Add a real Vitest test asserting .env.example contains no gittensory_<word>
metric-shaped string (matching the actual registered loopover_* metric families in src/selfhost/metrics.ts),
so this specific drift class — and not just these two instances — is caught going forward.

Expected Outcome

An operator following the QUEUE_CONCURRENCY tuning guidance in .env.example can paste the metric names it
gives directly into Grafana/Prometheus and get real data back.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions