Skip to content

fix: killed background task leaves a queryable trace, not a clean success (#2467) - #2473

Merged
vybe merged 2 commits into
devfrom
feature/2467-bg-kill-visibility
Sep 1, 2026
Merged

fix: killed background task leaves a queryable trace, not a clean success (#2467)#2473
vybe merged 2 commits into
devfrom
feature/2467-bg-kill-visibility

Conversation

@webmixgamer

Copy link
Copy Markdown
Contributor

Summary

  • A claude --print turn ending with a local_bash background task in flight recorded a clean success whose response was the model's announcement, while the CLI killed the task ~5s after exit — cost billed, work lost, zero signal (observed in production: a weekly data pull dead for 3 consecutive success fires). The kill events (task_updated {"status":"killed"} / task_notification {"status":"stopped"}) were already in the stored execution_log; nothing structured read them.
  • Fix is backend-side at terminal write, deliberately (the bug: schedule_executions.tool_calls stores a verbatim copy of execution_log — breaks the tool-call metric and survives log retention #1741 extract_tool_calls precedent): a new pure leaf services/execution_integrity.py scans the transcript inside apply_result and persists the structured record to a new nullable schedule_executions.turn_integrity column — so the fix reaches every deployed agent image with no base-image rebuild and no cold recreate, and the agent side stays byte-identical (pinned by permanent negative controls). This deviates from the issue's proposed agent-side tracker; every AC is satisfied and the deviation was approved at the plan gate.
  • Detection keys on the kill lifecycle events, never the ledger: the CLI drains background_tasks_changed to [] before exiting, so any snapshot-keyed counter reads 0 at finalize regardless of the denylist — the negative-control test proves the ledger-widening "fix" is inert. was_backgrounded_by: tool_timeout|requested distinguishes a foreground command the harness auto-promoted (always an incident) from deliberate fire-and-forget.
  • The same column carries the bug: turn ending with background subagents still in flight records the execution as SUCCESS and silently discards the work #2127 waited-path background_tasks_pending_at_exit count (issue root cause 3 — previously reported in metadata, persisted nowhere). A visible notice prefixes the stored response, the returned result, the feat: system-emitted agent.task.completed/failed events at execution terminal (async caller report-back) #1578 event summary and the channel completion report. NULL ≡ "no evidence", never "verified healthy" (the clone_status convention) — a mixed fleet needs no coordination.

Changes

  • src/backend/services/execution_integrity.py (new) — pure scan + notice builder. Privacy/containment: structural fields only (id/type/origin/status/end_time), charset-validated (^[A-Za-z0-9._:-]{1,128}$, failures replaced not dropped), closed enums, list capped at 20; description/summary/output_file/command structurally unreachable (bug: turn ending with background subagents still in flight records the execution as SUCCESS and silently discards the work #2127 privacy rule; forged-stream containment per fix(agent-runtime): prevent MCP stdio servers from polluting Claude's inherited stdout (#630 follow-up) #640 — a compromised agent can only false-flag its own runs, which its response text could already do).
  • src/backend/services/task_execution_service.pyapply_result SUCCESS branch derives + belt-try/excepts (a raise here would record a billed success as FAILED) + sanitize_execution_log boundary belt + notice prepend on the one sanitized_resp variable all downstream consumers share.
  • Column plumbing (Invariant Feature/vector log retention #3 dual-track): db/schema.py, db/tables.py, SQLite execution_turn_integrity migration, Alembic 0049_execution_turn_integrity (chained off 0048, ADD COLUMN IF NOT EXISTS per the fresh-PG convention).
  • Readers: tolerant row reader + ScheduleExecution + ExecutionSummary/ExecutionResponse/FleetExecutionSummary + both explicit list SELECTs (get_fleet_executions, get_agent_executions_summary — the second was caught in review; the 2026-08-21 every-reader learnings class).
  • src/backend/database.py — facade wrapper forwards the new kwarg. The gap was caught by the live E2E (first real terminal write raised unexpected keyword argument while every mocked-db test stayed green) and is now pinned by a signature-parity test + a learnings entry (kwarg-widening mode of the 2026-07-06 facade class).
  • db.update_execution_statusconditional kwarg (the retry_count pattern): an unconditional None would NULL the column on the FAILED→SUCCESS resurrect CAS.
  • Tests: tests/unit/test_2467_turn_integrity.py (33 — scan/notice/derive/apply_result golden kwargs/facade parity/real-engine reader parity) + tests/unit/test_2467_bg_kill_agent_negative_controls.py (4 — the renamed repro fixtures, now permanent agent-side negative controls). test_2127_background_task_gate.py untouched and green.
  • Docs: flow docs (parallel-headless-execution.md new section + revision entry; task-execution-service.md), architecture.md (service entry + schema), learnings entry, CSO diff report (0 findings).

Test Plan

  • pytest tests/unit/test_2467_turn_integrity.py tests/unit/test_2467_bg_kill_agent_negative_controls.py -v — 37 passed
  • Full unit suite: 13219 passed; 3 failures pre-existing (1× test_1920 enterprise-submodule allowlist — zero enterprise files in this diff; 2× known ordering flake bug(ci): test_1081_physical_meter ceiling-clamp test fails under full-suite ordering — settings_service caches a pre-eviction db #2045)
  • test_2127_background_task_gate.py (34) green untouched; test_1083_apply_result.py + CAS/terminal suites green
  • Alembic single head (check_alembic_heads: 50 revisions, 1 head 0049); test_schema_parity green
  • Live E2E on a dev instance: the incident-shape run stored status: success + turn_integrity with the full structured record on the fleet list, the notice prefixing the stored response, and the proof file absent (work genuinely lost, now visible). Healthy runs byte-identical.

Out of scope (documented residuals)

FAILED-row (502) kill capture (already honest failures; composable later at the #1853 salvage site) · chat path (claude_code.py, interactive) · UI consumption (ent#333 class) · recovered_terminal persistence · the sync /task HTTP passthrough body shows the agent's raw response (every stored/downstream surface carries the notice). Sibling issue #2468 (ScheduleWakeup offered in one-shot runs) removes the mechanism that talks the model into the plan; it stacks on this PR.

Fixes #2467

🤖 Generated with Claude Code

webmixgamer and others added 2 commits September 1, 2026 15:00
… a clean success (#2467)

A `claude --print` turn ending with a background shell in flight recorded
`success` with the model's announcement as the response while the CLI killed
the task ~5s after exit — cost billed, work lost, no signal anywhere. The
kill lifecycle events already ride `execution_log` on every deployed agent
image, so the fix derives turn-integrity flags backend-side at terminal
write (the #1741 no-rebuild precedent): a new pure leaf
`services/execution_integrity.py` scans the transcript inside `apply_result`
(keyed on the kill events, never the ledger — the CLI drains it to [] before
exiting, so snapshot-keyed counters read 0 by construction), persists the
structured record to new nullable `schedule_executions.turn_integrity`
(dual-track migration: SQLite `execution_turn_integrity` + Alembic 0049),
prepends a visible notice to the stored response, and surfaces the field on
ExecutionSummary, ExecutionResponse, FleetExecutionSummary and both explicit
list SELECTs. The same column carries the #2127 waited-path pending count
(previously reported in metadata, persisted nowhere). Records hold
structural fields only — id/type/origin/status, charset-validated, capped —
never description/command text (the #2127 privacy rule). NULL means "no
evidence", never "verified healthy". The agent side is byte-identical,
pinned by permanent negative controls; healthy runs are unchanged.

Fixes #2467

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…isibility

# Conflicts:
#	docs/memory/learnings.md

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated via /validate-pr: dual-track migration verified (SQLite execution_turn_integrity + Alembic 0049, single head, pg-migrations green); conditional-kwarg write guards the FAILED→SUCCESS resurrect CAS; facade signature parity pinned; 37 named regression tests + agent-side negative controls + live E2E on dev; CSO diff report 0 findings. Fixes #2467 closing keyword present — auto-promotes on merge.

@vybe
vybe merged commit efc6d7d into dev Sep 1, 2026
27 checks passed
@webmixgamer
webmixgamer deleted the feature/2467-bg-kill-visibility branch September 1, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants