Skip to content

fix(orb): ai-e2e-test-gen.ts and linked-issue-satisfaction-run.ts never set finalAttempt, so retried attempts log loud instead of quiet #8673

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

src/selfhost/ai.ts:936-939 computes const level = input.finalAttempt === false ? "warn" : "error";, with a doc comment explaining the contract: "only the FINAL attempt of a caller's own
retry loop is Sentry-visible (error); a retried attempt logs at warn ... a single-shot caller that
never sets this field keeps today's always-loud behavior."

src/services/ai-slop.ts:163, src/services/ai-review.ts:1228,2013, and
src/services/issue-plan-draft.ts:212 all correctly compute finalAttempt: attempt === LAST && modelIndex === LAST_MODEL on their multi-attempt ai.run() calls, honoring the contract.

src/services/ai-e2e-test-gen.ts:203-207 and src/services/linked-issue-satisfaction-run.ts:89-93
both run the identical 2-model × 3-attempt retry loop shape but never set finalAttempt in the
options object passed to ai.run. Per the contract above, an unset finalAttempt is reserved for
genuinely single-shot callers and always logs loud (error) — but neither of these two functions is
single-shot; both retry up to 6 times per invocation. As a result, every transient failure that
eventually succeeds on a later retry still logs a Sentry-visible error for each failed earlier
attempt in these two functions specifically, contrary to the documented intent that a retried
attempt should log at quiet warn.

This is an observability/noise bug (inflated Sentry error volume specifically for these two
features), not a functional one — the underlying retry/fallback behavior itself works correctly.

Requirements

  • Add finalAttempt: attempt === LAST_ATTEMPT && modelIndex === LAST_MODEL_INDEX (or the equivalent
    computation matching this file's own loop variables) to both ai-e2e-test-gen.ts's and
    linked-issue-satisfaction-run.ts's ai.run() call options, mirroring the pattern already used by
    ai-slop.ts/ai-review.ts/issue-plan-draft.ts exactly.

Deliverables

  • ai-e2e-test-gen.ts's ai.run() call sets finalAttempt correctly for every attempt/model
    combination in its retry loop.
  • linked-issue-satisfaction-run.ts's equivalent call has the same fix.
  • A new test for each function stubs env.AI.run to fail once then succeed, and asserts the
    failing call's logged level is warn (i.e. finalAttempt: false was passed) — pinning the
    same convention ai-slop.ts's existing test already pins.
  • A new test for each function confirms the genuinely final failed attempt (all models/attempts
    exhausted) logs at error (finalAttempt: true).

All four Deliverables are required in the same PR.

Test Coverage Requirements

src/** is measured by codecov/patch (99%+ target, branch-counted). The new tests must exercise
both the non-final (warn) and final (error) branches for both functions, matching
ai-slop.ts's existing test pattern for the same distinction.

Expected Outcome

ai-e2e-test-gen.ts and linked-issue-satisfaction-run.ts correctly log a retried (non-final)
attempt's failure at quiet warn, matching every other multi-attempt AI-calling function in the
codebase, eliminating inflated Sentry error-volume noise for these two features specifically.

Links & Resources

  • src/selfhost/ai.ts:936-939 (the documented finalAttempt contract)
  • src/services/ai-e2e-test-gen.ts:203-207, src/services/linked-issue-satisfaction-run.ts:89-93
    (the two functions to fix)
  • Already-correct siblings: src/services/ai-slop.ts:163, src/services/ai-review.ts:1228,2013,
    src/services/issue-plan-draft.ts:212

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions