Skip to content

fix(orb): ai-e2e-test-gen.ts's per-model retry loop burns all attempts against an already rate-limited model #8672

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

src/services/ai-e2e-test-gen.ts:195's own doc comment says it "mirrors runWorkersSlopOpinion's
exact shape." Its per-model retry loop's catch block (lines 200-214, specifically line 210,
catch { /* retry / fall through to fallback */ }) has no rate-limit check at all —
isRateLimitError is never imported anywhere in this file.

Every other near-identical per-model retry loop in the codebase correctly short-circuits on a rate
limit:

  • src/services/ai-slop.ts:169-174} catch (error) { if (isRateLimitError(error)) break; }
  • src/review/planner.ts:137
  • src/services/issue-plan-draft.ts:219
  • src/services/linked-issue-satisfaction-run.ts:98

isRateLimitError's own doc comment (src/services/ai-review.ts:1109-1120) states it exists
specifically so "every independent AI-calling retry loop ... can share this one definition," and
that retrying the same model against a live 429 "has near-zero chance of success."

Consequence: when the default AI provider 429s while generating an E2E test,
ai-e2e-test-gen.ts burns all E2E_TEST_GEN_ATTEMPTS_PER_MODEL (3) retries against the same
rate-limited model — using half of its total pre-budgeted E2E_TEST_GEN_MAX_CALLS (6) on attempts
documented elsewhere in this same codebase as having zero chance of success — before ever reaching
the fallback model, adding avoidable latency and wasted quota.

Requirements

  • Import isRateLimitError into ai-e2e-test-gen.ts and add the same break on a detected rate
    limit that every sibling retry loop already has, mirroring ai-slop.ts:169-174 exactly.
  • Do not change behavior for any non-rate-limit failure (those should continue to retry as today).

Deliverables

  • ai-e2e-test-gen.ts's per-model retry loop breaks out of retrying the same model immediately
    on a detected rate-limit error, moving to the fallback model instead of burning all 3
    attempts.
  • A new test stubs env.AI.run to throw a rate-limit-shaped error for the first model on every
    call, and asserts the first model is called at most once before the second (fallback) model is
    attempted — mirroring ai-slop.ts's existing rate-limit test.
  • A regression test confirms a non-rate-limit failure still retries the full configured attempt
    count for the same model (proving the fix doesn't over-broaden the break condition).

All three Deliverables are required in the same PR.

Test Coverage Requirements

src/** is measured by codecov/patch (99%+ target, branch-counted). Both new tests must exercise
the rate-limit and non-rate-limit branches directly, matching ai-slop.ts's existing test pattern
for the same distinction.

Expected Outcome

ai-e2e-test-gen.ts's retry behavior on a rate-limited model matches every other AI-calling retry
loop in the codebase: immediate fallback on a detected rate limit, full retry budget preserved for
genuinely retryable failures.

Links & Resources

  • src/services/ai-e2e-test-gen.ts:195-214 (the retry loop to fix)
  • src/services/ai-slop.ts:169-174 (the already-correct pattern to mirror)
  • src/services/ai-review.ts:1109-1120 (isRateLimitError, the shared helper to reuse)
  • Other correct siblings: src/review/planner.ts:137, src/services/issue-plan-draft.ts:219,
    src/services/linked-issue-satisfaction-run.ts:98

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions