Skip to content

orb(webhook): ~25 webhooks per review — the comment idempotency guard never fires because the panel embeds a per-pass timestamp #9069

Description

@JSONbored

ORB has ingested ~309,600 webhook deliveries lifetime against ~12,500 reviews — roughly 25 webhooks per review. A large share is traffic ORB generates against itself, and the guard written specifically to prevent it is defeated by its own payload.

Live volume (all time)

event action count share
check_suite completed 106,943 35%
issue_comment edited 79,612 26%
issue_comment created 28,411 9%
pull_request labeled 14,555 5%
check_suite requested 11,134 4%
pull_request opened / closed 17,186 6%
pull_request assigned 7,510 2%
issues labeled 6,862 2%
pull_request review_requested 6,175 2%

Last 24h is the same shape: check_suite.completed 865, issue_comment.created 410, issue_comment.edited 331.

Root cause of the #2 event: the no-op guard can never fire

src/github/comments.ts ~140-147 exists precisely to stop this, and says so:

skip the PATCH when the rendered body is byte-identical to what's already posted. The re-gate sweep re-renders the same surface every cycle for an unchanged PR; without this, every cycle PATCHes GitHub (a write + rate-limit cost) for no visible change.

It compares canonical.body === body. But the body contains a per-pass timestamp:

  • src/review/unified-comment.ts ~759 — <sub>Review updated: ${reviewTimestamp}</sub>
  • src/review/unified-comment-bridge.ts ~992 — reviewedAt: args.reviewedAt ?? new Date(), and the processor never passes reviewedAt

So every re-render produces a body differing by at least the timestamp → the equality check always failsevery pass PATCHes. Each PATCH is a GitHub API write (rate-limit spend) and generates an inbound issue_comment.edited delivery that ORB then signature-verifies, hashes, dedups, DB-inserts, classifies as self-authored noise, and discards.

That is ~80k round-trips of pure self-inflicted work — 26% of all ingress — for a changing timestamp.

Secondary amplifier: ignored noise is still persisted

enqueueVerifiedWebhook (src/github/webhook.ts ~178-181) correctly declines to enqueue noise, but still writes a webhook_events row with status: "processed". So check_suite.requested (11k, filtered by isNonCompletedCiWebhook) and the bot comment-edits (filtered by isBotAuthoredIssueCommentEditWebhook) all inflate the table and make "processed" useless as a work metric — 303k "processed" is emphatically not 303k units of work.

Fix, in order of leverage

  1. Exclude the timestamp from the idempotency comparison. Normalize the Review updated: line out of both sides before canonical.body === body, or only re-stamp reviewedAt when some other part of the body actually changed. Expected effect: eliminates most of the 80k self-generated edits, plus the matching outbound writes and their rate-limit cost. Highest leverage, smallest change.
  2. Unsubscribe from events we filter at ingress anyway. check_suite.requested, pull_request.assigned, issues.milestoned, and the bot comment-edit class are dropped on arrival — unsubscribing stops them at GitHub, saving ingress, signature verification, DB writes, and rate-limit headroom. (Coordinate with orb(webhook): install health reports 'healthy' without check_run/check_suite, and the setup wizard ships a manifest missing two required events #9058: subscriptions must be derived from REQUIRED_INSTALLATION_EVENTS, and check_run/check_suite.completed must stay.)
  3. Do not persist ignored noise, or persist it under a distinct status: "ignored" so processed means "did work". This also makes the stuck-queued metric in orb(webhook): 5,547 deliveries stuck 'queued' are permanently un-redeliverable — 4,900 are check_suite.completed, the auto-merge trigger #9054 legible.
  4. Add a per-PR webhook-volume metric so a future feedback loop is visible immediately rather than discovered at 300k.

Acceptance

  • An unchanged PR re-gated by the sweep produces zero outbound PATCHes and therefore zero issue_comment.edited deliveries.
  • webhook_events distinguishes ignored from processed.
  • Webhooks-per-review drops substantially from ~25.

Refs #9054 (stuck queued), #9058 (event subscriptions).

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions