Skip to content

orb(queue): foreground-liveness cancels every deliberate delay, not just rate-limit deferrals — the contributor's promised close grace window collapses from 300s to 60s #9127

Description

@JSONbored

Problem

The foreground-liveness sweep releases every future-scheduled foreground job, not just jobs deferred
by rate-limit admission. It cancels deliberate delays — including the grace window ORB promises a
contributor before closing their PR.

releaseStaleForegroundDeferrals (src/selfhost/pg-queue.ts:706-748) selects candidates with no
provenance filter whatsoever:

SELECT id, payload, created_at FROM ${TABLE}
 WHERE status='pending' AND priority>=$1 AND run_after>$2 ORDER BY created_at ASC, id ASC LIMIT $3

Any pending foreground row scheduled in the future qualifies. Eligibility is then:

732:  const ageStale = isForegroundDeferralStale(foregroundLivenessConfig, pendingSinceMs, now);
733:  const rateLimitClear = await isRateLimitAdmissionNowClear(row.payload, admissionCache);
734:  if (!ageStale && !rateLimitClear) continue;

so rateLimitClear alone releases a row — no age requirement. And isRateLimitAdmissionNowClear
returns true outright for any job with no rate-limit bucket:

671:  const target = githubRateLimitAdmissionTargetForJob(message);
672:  if (target === null) return true;

githubRateLimitAdmissionTargetForJob (src/selfhost/queue-common.ts:521-548) returns a target only for
github-webhook, non-sweep agent-regate-pr, and background-budget jobs. recapture-preview returns
null
— so it is always "clear" and always released. The release is
UPDATE … SET run_after=$1 (= now).

The module header's premise — that only rate-limit admission pushes a foreground run_after into the
future — is false: enqueue(message, delaySeconds) sets run_after = now + delaySeconds*1000 for
foreground types too.

Trigger — the contributor grace window

src/queue/processors.ts:3649-3665 implements the linked-issue flag-then-close grace period using exactly
that job type:

const delaySeconds = Math.max(0, linkedIssueRulesConfig.closeDelaySeconds);
const verifyJob = { type: "recapture-preview" as const, deliveryId: `linked-issue-verify:...` , ... };
await (delaySeconds > 0 ? env.JOBS.send(verifyJob, { delaySeconds }) : env.JOBS.send(verifyJob))

closeDelaySeconds goes up to 300s (src/openapi/schemas.ts:832), and the flag comment tells the
contributor they have ~${closeDelaySeconds}s (src/settings/agent-actions.ts:1340). The sweep runs
every checkIntervalMs (60s default) and releases up to 25 rows per tick — so Pass 2 fires within ≤60s
and closes the PR. The contributor is told 300 seconds and gets 60.

Other victims of the same mechanism:

  • processors.ts:11671-11681PREVIEW_POLL_SECONDS = 90 visual-preview backoff collapses to ≤60s,
    burning MAX_PREVIEW_POLL_ATTEMPTS before the preview deploy is live. Each attempt is a full
    re-review plus a headless-Chromium capture.
  • processors.ts:4553, :4596 — CI-coalesce and mergeable-state trailing re-reviews run before their
    window closes; their transient dedupe key is already claimed, so no retry is scheduled.
  • Every sweep fan-out stagger (Math.min(index * 10, 600), 8 call sites) compresses to 25/min,
    nullifying the GitHub-REST and AI-spend staggering it exists to provide.

Impact

Wrong-close. A contributor's remediation window silently shrinks up to 5×, and the PR is closed
one-shot. Secondarily: designed-in rate/spend staggers are cancelled, and trailing re-checks read the
state they were explicitly scheduled to wait past.

Requirements

  • Release only deferrals authored by an admission gate. Add a deferred_by column set at the
    deferral site (the rate-limit paths already write a recognisable last_error at pg-queue.ts:1164,
    which is a viable interim key), and never release a row whose run_after came from the enqueue's own
    delaySeconds.
  • Correct the module header, which currently states the false premise this bug rests on.
  • Mirror the fix in src/selfhost/sqlite-queue.ts:385-421 (verified twin).
  • Add an invariant test: a job enqueued with delaySeconds: N is not runnable before N seconds,
    regardless of liveness-sweep activity.

Test Coverage Requirements

99%+ patch coverage, branch-counted. Both arms of the provenance check, plus a regression test asserting
the linked-issue grace window survives a liveness sweep.

Links & Resources

Boundaries

Deferral-provenance only. No change to the liveness sweep's rate-limit-release purpose, which is correct
and should keep working.

maintainer-only — scheduler correctness on a close-critical path.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions