Re-queue worker pods on transient 502/503/504 from the Kubernetes API

[1fanwang]

Why

Inspired from @jscheffl 's comment #68480 (comment)
When KubernetesExecutor fails to create a worker pod, it re-queues the task on an
exceeded-quota / stale-version conflict, 500, or 429, but lets 502 / 503 / 504
fall through and fail the task immediately. Those are exactly the gateway / unavailable /
timeout codes that spike when the API server or an admission webhook is under load. Concretely,
the API server returns 503 with Retry-After: 1 on every graceful shutdown (rollout, node
drain, control-plane autoscale — see apiserver waitgroup.go), so a worker pod CREATE that is
in flight during any apiserver roll currently kills the task outright instead of retrying.
client-go retries all 5xx and honours Retry-After; the executor does not.

What

Add 502 / 503 / 504 to the transient set, so a task that hits one is re-queued (bounded by
task_publish_max_retries) instead of failed. When the response carries Retry-After (a
shutting-down apiserver always sends it on 503), pause pod creation until then via
create_pods_after, exactly as the 429 path already does; other transient 5xx without the
header retry on the next scheduler loop like 500. The Retry-After parse is guarded so a
malformed value can't crash the scheduler loop.

This gap was noticed while reviewing #68480 (concurrent pod creation). It is pre-existing on the
current sync path and independent of that change, so it goes as its own PR; the two touch the same
error handler, so whichever lands second will need a trivial rebase.

Tests

test_run_next_exception_requeue gains 502 / 504 (re-queue, retry next loop), 503 + Retry-After
(re-queue, loop backs off), and 503 with retries exhausted (fails). Each fails without this change.

E2E

Live kind (real API server, real KubernetesExecutor.sync()), with a proxy injecting
503 + Retry-After: 1 on the first CREATE per task — identical config (task_publish_max_retries=3),
only the fix differs:

Run (retries=3)	behaviour	pods created
before (main)	503 → task failed	0 / 6
after (this PR)	503 → re-queued, retried	6 / 6

Raw — after

loop 0..5: queue 6->6 failed_events=0     # each loop: one task hits 503+Retry-After, re-queues, loop backs off
loop 6:    queue 6->0 failed_events=0     # backoff elapsed -> retries succeed, all pods created
FINAL failed_events=0 queue_remaining=0
# injector: INJECT 503 (Retry-After=1) x6 (first attempt) -> PASS 201 x6 (retry)
# cluster: retrydag-t000..t005  Running

Risk

Behaviour change on the default path: a 502 / 503 / 504 now re-queues instead of failing the
task — bounded by task_publish_max_retries (default 0, i.e. inert unless retries are already
enabled). The existing 429 / 500 / quota / conflict handling is unchanged.