fix: killed background task leaves a queryable trace, not a clean success (#2467) - #2473
Merged
Conversation
… a clean success (#2467) A `claude --print` turn ending with a background shell in flight recorded `success` with the model's announcement as the response while the CLI killed the task ~5s after exit — cost billed, work lost, no signal anywhere. The kill lifecycle events already ride `execution_log` on every deployed agent image, so the fix derives turn-integrity flags backend-side at terminal write (the #1741 no-rebuild precedent): a new pure leaf `services/execution_integrity.py` scans the transcript inside `apply_result` (keyed on the kill events, never the ledger — the CLI drains it to [] before exiting, so snapshot-keyed counters read 0 by construction), persists the structured record to new nullable `schedule_executions.turn_integrity` (dual-track migration: SQLite `execution_turn_integrity` + Alembic 0049), prepends a visible notice to the stored response, and surfaces the field on ExecutionSummary, ExecutionResponse, FleetExecutionSummary and both explicit list SELECTs. The same column carries the #2127 waited-path pending count (previously reported in metadata, persisted nowhere). Records hold structural fields only — id/type/origin/status, charset-validated, capped — never description/command text (the #2127 privacy rule). NULL means "no evidence", never "verified healthy". The agent side is byte-identical, pinned by permanent negative controls; healthy runs are unchanged. Fixes #2467 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…isibility # Conflicts: # docs/memory/learnings.md
vybe
approved these changes
Sep 1, 2026
vybe
left a comment
Contributor
There was a problem hiding this comment.
Validated via /validate-pr: dual-track migration verified (SQLite execution_turn_integrity + Alembic 0049, single head, pg-migrations green); conditional-kwarg write guards the FAILED→SUCCESS resurrect CAS; facade signature parity pinned; 37 named regression tests + agent-side negative controls + live E2E on dev; CSO diff report 0 findings. Fixes #2467 closing keyword present — auto-promotes on merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
claude --printturn ending with alocal_bashbackground task in flight recorded a cleansuccesswhose response was the model's announcement, while the CLI killed the task ~5s after exit — cost billed, work lost, zero signal (observed in production: a weekly data pull dead for 3 consecutivesuccessfires). The kill events (task_updated {"status":"killed"}/task_notification {"status":"stopped"}) were already in the storedexecution_log; nothing structured read them.extract_tool_callsprecedent): a new pure leafservices/execution_integrity.pyscans the transcript insideapply_resultand persists the structured record to a new nullableschedule_executions.turn_integritycolumn — so the fix reaches every deployed agent image with no base-image rebuild and no cold recreate, and the agent side stays byte-identical (pinned by permanent negative controls). This deviates from the issue's proposed agent-side tracker; every AC is satisfied and the deviation was approved at the plan gate.background_tasks_changedto[]before exiting, so any snapshot-keyed counter reads 0 at finalize regardless of the denylist — the negative-control test proves the ledger-widening "fix" is inert.was_backgrounded_by: tool_timeout|requesteddistinguishes a foreground command the harness auto-promoted (always an incident) from deliberate fire-and-forget.background_tasks_pending_at_exitcount (issue root cause 3 — previously reported in metadata, persisted nowhere). A visible notice prefixes the stored response, the returned result, the feat: system-emitted agent.task.completed/failed events at execution terminal (async caller report-back) #1578 event summary and the channel completion report. NULL ≡ "no evidence", never "verified healthy" (theclone_statusconvention) — a mixed fleet needs no coordination.Changes
src/backend/services/execution_integrity.py(new) — pure scan + notice builder. Privacy/containment: structural fields only (id/type/origin/status/end_time), charset-validated (^[A-Za-z0-9._:-]{1,128}$, failures replaced not dropped), closed enums, list capped at 20;description/summary/output_file/commandstructurally unreachable (bug: turn ending with background subagents still in flight records the execution as SUCCESS and silently discards the work #2127 privacy rule; forged-stream containment per fix(agent-runtime): prevent MCP stdio servers from polluting Claude's inherited stdout (#630 follow-up) #640 — a compromised agent can only false-flag its own runs, which its response text could already do).src/backend/services/task_execution_service.py—apply_resultSUCCESS branch derives + belt-try/excepts (a raise here would record a billed success as FAILED) +sanitize_execution_logboundary belt + notice prepend on the onesanitized_respvariable all downstream consumers share.db/schema.py,db/tables.py, SQLiteexecution_turn_integritymigration, Alembic0049_execution_turn_integrity(chained off0048,ADD COLUMN IF NOT EXISTSper the fresh-PG convention).ScheduleExecution+ExecutionSummary/ExecutionResponse/FleetExecutionSummary+ both explicit list SELECTs (get_fleet_executions,get_agent_executions_summary— the second was caught in review; the 2026-08-21 every-reader learnings class).src/backend/database.py— facade wrapper forwards the new kwarg. The gap was caught by the live E2E (first real terminal write raisedunexpected keyword argumentwhile every mocked-db test stayed green) and is now pinned by a signature-parity test + a learnings entry (kwarg-widening mode of the 2026-07-06 facade class).db.update_execution_status— conditional kwarg (theretry_countpattern): an unconditionalNonewould NULL the column on the FAILED→SUCCESS resurrect CAS.tests/unit/test_2467_turn_integrity.py(33 — scan/notice/derive/apply_result golden kwargs/facade parity/real-engine reader parity) +tests/unit/test_2467_bg_kill_agent_negative_controls.py(4 — the renamed repro fixtures, now permanent agent-side negative controls).test_2127_background_task_gate.pyuntouched and green.parallel-headless-execution.mdnew section + revision entry;task-execution-service.md),architecture.md(service entry + schema), learnings entry, CSO diff report (0 findings).Test Plan
pytest tests/unit/test_2467_turn_integrity.py tests/unit/test_2467_bg_kill_agent_negative_controls.py -v— 37 passedtest_1920enterprise-submodule allowlist — zero enterprise files in this diff; 2× known ordering flake bug(ci): test_1081_physical_meter ceiling-clamp test fails under full-suite ordering — settings_service caches a pre-eviction db #2045)test_2127_background_task_gate.py(34) green untouched;test_1083_apply_result.py+ CAS/terminal suites greencheck_alembic_heads: 50 revisions, 1 head0049);test_schema_paritygreenstatus: success+turn_integritywith the full structured record on the fleet list, the notice prefixing the stored response, and the proof file absent (work genuinely lost, now visible). Healthy runs byte-identical.Out of scope (documented residuals)
FAILED-row (502) kill capture (already honest failures; composable later at the #1853 salvage site) · chat path (
claude_code.py, interactive) · UI consumption (ent#333 class) ·recovered_terminalpersistence · the sync/taskHTTP passthrough body shows the agent's raw response (every stored/downstream surface carries the notice). Sibling issue #2468 (ScheduleWakeup offered in one-shot runs) removes the mechanism that talks the model into the plan; it stacks on this PR.Fixes #2467
🤖 Generated with Claude Code