Worker health: cold-start grace period - #30
tydinjarman wants to merge 2 commits into
Conversation
A freshly launched worker boots silently and streams no evidence until it starts working, so it cannot be meaningfully judged stuck or off track yet. While the active worker is younger than the configured grace window, worker-health defers judgment with CONTINUE instead of steering or stopping it. New typed config: cold_start_grace_seconds (default 0.0, disabled; FOREMAN_COLD_START_GRACE_SECONDS), wired through the worker-health responsibility settings. A worker with no recorded start time is not treated as young: fail closed toward the established behavior.
JARosen
left a comment
There was a problem hiding this comment.
Cold-start hysteresis belongs in core.worker-health, but the current condition is broader than the rationale. It suppresses both worker_stuck and work_off_track solely based on worker age. A young worker that has already produced strong evidence of going in the wrong direction would therefore be allowed to continue.
Please narrow this to the actual cold-start case—for example, defer stuckness while the worker is young and has produced little or no activity—without suppressing evidence-backed off-track intervention. A policy-level test combining age, output or activity, and a high off-track score would help establish the intended safety boundary.
JARosen's review noted the grace suppressed both worker_stuck and work_off_track solely based on worker age, letting a young worker with strong off-track evidence continue. Narrowed semantics: - work_off_track is never deferred by cold-start grace: an evidence-backed off-track signal always intervenes regardless of worker age. - worker_stuck is deferred only while the worker is young (within cold_start_grace_seconds) AND quiet, where quiet means captured stdout+stderr at most cold_start_quiet_output_bytes (new knob, default 0, env FOREMAN_COLD_START_QUIET_OUTPUT_BYTES). - A worker with no recorded start time is not treated as young (fails closed toward established behavior). Tests: 180 passed; new policy-level tests cover off-track firing for a young worker with high off-track score, stuckness deferral only for a young quiet worker, and the quiet-output-bytes boundary.
tydinjarman
left a comment
There was a problem hiding this comment.
Addressed the cold-start review feedback (pushed as commit 16e0676).
The grace now applies only to worker_stuck and only while the worker is
young and quiet — captured stdout+stderr at most
cold_start_quiet_output_bytes (new knob, default 0, env
FOREMAN_COLD_START_QUIET_OUTPUT_BYTES). An evidence-backed
work_off_track signal always intervenes regardless of worker age; a young
worker that has produced strong evidence of going the wrong way is steered or
stopped immediately. A worker with no recorded start time fails closed toward
the established behavior, and when both checks fire outside grace, the
evidence-backed off-track verdict now takes precedence over the
max-confidence tie-break.
Test evidence: full suite 180 passed, including 4 new policy-level tests —
off-track never suppressed for a young quiet worker, off-track still fires
for a young worker with output and a high off-track score (your
safety-boundary case), activity disqualifying stuckness deferral, and the
quiet-output-bytes boundary. Ruff clean.
JARosen
left a comment
There was a problem hiding this comment.
Thank you for being so responsive and for narrowing the grace period to young, quiet stuckness. The revised off-track behavior addresses the original safety concern.
One cross-responsibility arbitration issue remains. Deferred stuckness currently proposes a priority-900 CONTINUE carrying the stuck confidence. Because policy breaks equal-priority ties by confidence, that deferral can suppress another responsibility actionable directive. I reproduced a 0.95 cold-start stuck score selecting CONTINUE over a 0.85 repository.instructions STEER_WORKER proposal.
Please make cold-start deferral abstain from proposing a directive, or use a priority that cannot override actionable responsibility output. If no other responsibility fires, the policy already supplies its ordinary priority-zero CONTINUE. Please add a policy-level regression test combining deferred stuckness with repository drift.
This remaining change is small. If you would prefer that I make it, say so and I am happy to help rather than taking over the branch without asking.
Supersedes part of #19, per your review: the "cold-start handling belongs in worker-health behavior" change, rebuilt from current
mainthrough the responsibility architecture, with no helper scripts.What it does
WorkerHealthResponsibility.directivesnow checks_within_cold_start_grace(state)after the stuck/off-track thresholds are met: while the active worker is younger thancold_start_grace_seconds, it emitsCONTINUE("active worker is within the cold-start grace period", priority 900) instead ofSTEER_WORKER/STOP_WORKER.FactoryConfigfieldcold_start_grace_seconds(default0.0= disabled, preserving upstream behavior;FOREMAN_COLD_START_GRACE_SECONDS), plumbed through thecore.worker-healthresponsibility settings in_SETTING_FIELDS.Validation
tests/test_worker_health_cold_start.py; full suite 176 passed,ruff checkclean.builtin.pyin disjoint regions, but if merged in sequence the later ones may need a trivial rebase — happy to rebase on request.