Skip to content

feature(ops): auto-resolve PagerDuty incidents once ops anomalies clear #8903

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

src/review/ops-wire.ts:304-306 (inside runOpsAlerts) explicitly documents: "This does not yet send a matching 'resolve' event once anomalies clear (would need tracking previous-tick state) -- an operator currently resolves the incident manually once the underlying condition is fixed." Confirmed no other code path sends a resolve event for dedupKey: ops_anomaly:${repoFullName}.

Requirements

Persist previous-tick anomaly state per repo (e.g. in the same table detectOutcomeAnomalies reads) and call the PagerDuty trigger function with an eventAction: "resolve" (or equivalent) when a repo that had anomalies last tick has none this tick.

Deliverables

  • Previous-tick anomaly state is persisted per repo
  • runOpsAlerts sends a PagerDuty resolve event when a repo's anomalies clear between ticks
  • A test simulating two consecutive ticks (anomaly present, then cleared), asserting a resolve event is sent on the second tick

All of the above Deliverables are required in the same PR unless the deliverable text itself states otherwise.

Test Coverage Requirements

src/** -- 99%+ patch coverage, branch-counted, on the new state-tracking and resolve-event path.

Expected Outcome

A PagerDuty incident for an ops anomaly is automatically resolved once the underlying condition clears, instead of requiring manual resolution every time.

Links & Resources

  • src/review/ops-wire.ts:304-306

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions