Skip to content

Blocked compiler versions disable every workflow in a repo with no notification path #59600

Description

@Calidus

Summary

When a compiler version lands in blockedVersions, every compiled workflow in a repository starts failing at activation — and there is no notification path, because the built-in failure reporting lives downstream of the job that just failed. The result is a silent, repo-wide outage that can persist indefinitely.

We lost several days to this. Filing because the mechanism guarantees that anyone it happens to will also not be told.

What happened

Every agentic workflow in our repository was compiled with v0.84.3. On 2026-09-03, github/gh-aw-actions commit 9271a18 changed .github/aw/compat.json from "blockedVersions": [] to a 23-entry list spanning v0.82.8–v0.85.3.

From the next activation onward, every run failed with:

##[error]Blocked compile-agentic version: v0.84.3 is in the blocked versions list.
Update gh-aw to the latest version and recompile your workflow.

The error message is clear and correct. The problem is that nobody sees it.

Why nothing reported it

activation fails, so every downstream job is skipped — including conclusion, which is where handle_agent_failure.cjs runs:

pre_activation  ->  success
activation      ->  failure     <- version check fails here
agent           ->  skipped
detection       ->  skipped
safe_outputs    ->  skipped
conclusion      ->  skipped     <- failure-issue handler lives here, never runs

Across every blocked run, zero failure issues were created. The feature designed to report this class of problem is structurally incapable of reporting this particular one.

Self-monitoring can't close this gap: any mechanism inside the compiled workflow is downstream of the check that disables the compiled workflow.

Why this is worse than an ordinary failure

Three properties compound:

  1. It is remote and evaluated per run. No commit, no config change, no upgrade on the consumer side. A repo that was green yesterday is dead today, so nobody has a recent change to suspect.
  2. It is repo-wide and simultaneous. Every workflow dies at once, so any workflow you rely on to tell you something is broken is also broken. In our case the affected set included our build-failure investigator — the outage suppressed the alarm for itself and for unrelated CI failures.
  3. The absence is the only symptom. GitHub Actions shows failure conclusions, but for status/schedule-triggered workflows the actor is a bot, so no human is notified. We discovered it by chance, when someone noticed a CI failure that should have produced a Slack post and didn't.

Proposed solutions

In rough order of value per effort:

1. Have the activation-stage check open a deduplicated issue before it exits. The step already detects the exact condition, already has a token, and already knows the workflow name. Creating or updating a single parent issue per repo ("Workflows blocked: compiled with v0.84.3, which is on the blocked list") turns a silent outage into a visible one and reuses the dedup logic that already exists for agent-failure issues. This is the smallest change that actually fixes it.

2. Warn before blocking. minRecommendedVersion already exists as a soft signal but only produces a log line nobody reads. A "will be blocked on " or "deprecating" tier in compat.json, surfaced as a job annotation or issue while runs still succeed, would let repos upgrade before the hard block rather than after.

3. Ship a non-agentic watchdog template. Something addable via gh aw add that runs as a plain Actions job — no activation, no agent — and checks the repo's compiled version against compat.json, optionally alongside gh aw health. The existing Agent Job Health Monitor doesn't serve here, since it is itself an agentic workflow and gets blocked with everything else.

Environment

  • Affected compiler version: v0.84.3
  • Now on: v0.88.7
  • Private repository with multiple agentic workflows

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions