Summary
services/backlog_service.py:240 lazily imports _execute_task_background from routers.chat, but that function no longer exists in routers/chat.py. Every backlog drain attempt fails with an ImportError, marks the queued execution as failed, and leaves the persistent task backlog (BACKLOG-001) silently non-functional. Observed on a live instance with a fan-out workload — 23 backlog drain failures in 24 hours, all with the same import error.
Component
Backend / Backlog Service (BACKLOG-001)
Priority
P1
Error
Backlog drain spawn failed: cannot import name '_execute_task_background' from 'routers.chat' (/app/routers/chat.py)
Location
- File:
services/backlog_service.py
- Line: 240
- Function:
BacklogService._spawn_drain
Root Cause
_spawn_drain does:
from routers.chat import _execute_task_background
grep -rn 'def _execute_task_background\|def execute_task_background' /app/ finds zero matches in routers/chat.py. The function appears to have been renamed or moved (likely during the work tracked in #95 "Route async task mode through TaskExecutionService") without updating this lazy import. The exception is caught at backlog_service.py:222-228, so the only surface signal is the per-execution Backlog drain spawn failed: ... error in schedule_executions.error.
The drain is invoked from BacklogService.on_slot_released (called by SlotService on slot free) and from periodic maintenance, so every queued task that should drain hits this error.
Net behavior: BACKLOG-001 is dead — any execution that gets queued to the backlog stays queued until it gets pulled and immediately fails on drain. Sync-mode chat_with_agent callers don't see this because they bypass the backlog (see related issue), but any async-mode caller relies on this path.
Reproduction Steps
- Have an agent at full slot capacity (e.g.,
max_parallel_tasks=3 and 3 active executions)
- Submit an async-mode parallel chat request to that agent (
async_mode=true, parallel=true)
- The chat router enqueues the request to the backlog — verify with
SELECT * FROM schedule_executions WHERE status='queued'
- Wait for a slot to free (or trigger a periodic drain)
- Check
schedule_executions.error for the dropped execution — observe Backlog drain spawn failed: cannot import name '_execute_task_background' from 'routers.chat'
Suggested Fix
Locate the renamed/moved function and update the lazy import. Search hits for the old name in the deployed image:
services/backlog_service.py:19: Drain imports `_execute_task_background` lazily to avoid a circular import
services/backlog_service.py:141: Spawn `_execute_task_background` on the reconstituted request.
services/backlog_service.py:240: from routers.chat import _execute_task_background
services/backlog_service.py:258: _execute_task_background(...)
routers/internal.py:267: asyncio.create_task(_execute_task_internal_background(...))
routers/internal.py:306: async def _execute_task_internal_background(task_service, request: ...)
Likely candidate: route through TaskExecutionService directly (per #95), reconstructing the same call shape used by the chat router's async path. The signature _spawn_drain builds includes release_slot=True, user_id, user_email, subscription_id, task_activity_id, collaboration_activity_id, x_source_agent — whichever replacement is wired up needs to accept those (or an equivalent context object).
A regression test that exercises the slot-release → drain path would prevent recurrence; the current exception-swallowing makes this kind of break silent.
Environment
- Trinity version:
c3f3f1c
- Python: 3.x (deployed image)
- Observed on a live instance with a fan-out caller dispatching 4-7 task batches every 1-5 min to 7 worker agents
Related
Summary
services/backlog_service.py:240lazily imports_execute_task_backgroundfromrouters.chat, but that function no longer exists inrouters/chat.py. Every backlog drain attempt fails with anImportError, marks the queued execution as failed, and leaves the persistent task backlog (BACKLOG-001) silently non-functional. Observed on a live instance with a fan-out workload — 23 backlog drain failures in 24 hours, all with the same import error.Component
Backend / Backlog Service (BACKLOG-001)
Priority
P1
Error
Location
services/backlog_service.pyBacklogService._spawn_drainRoot Cause
_spawn_draindoes:grep -rn 'def _execute_task_background\|def execute_task_background' /app/finds zero matches inrouters/chat.py. The function appears to have been renamed or moved (likely during the work tracked in #95 "Route async task mode through TaskExecutionService") without updating this lazy import. The exception is caught atbacklog_service.py:222-228, so the only surface signal is the per-executionBacklog drain spawn failed: ...error inschedule_executions.error.The drain is invoked from
BacklogService.on_slot_released(called bySlotServiceon slot free) and from periodic maintenance, so every queued task that should drain hits this error.Net behavior: BACKLOG-001 is dead — any execution that gets queued to the backlog stays queued until it gets pulled and immediately fails on drain. Sync-mode
chat_with_agentcallers don't see this because they bypass the backlog (see related issue), but any async-mode caller relies on this path.Reproduction Steps
max_parallel_tasks=3and 3 active executions)async_mode=true, parallel=true)SELECT * FROM schedule_executions WHERE status='queued'schedule_executions.errorfor the dropped execution — observeBacklog drain spawn failed: cannot import name '_execute_task_background' from 'routers.chat'Suggested Fix
Locate the renamed/moved function and update the lazy import. Search hits for the old name in the deployed image:
Likely candidate: route through
TaskExecutionServicedirectly (per #95), reconstructing the same call shape used by the chat router's async path. The signature_spawn_drainbuilds includesrelease_slot=True,user_id,user_email,subscription_id,task_activity_id,collaboration_activity_id,x_source_agent— whichever replacement is wired up needs to accept those (or an equivalent context object).A regression test that exercises the slot-release → drain path would prevent recurrence; the current exception-swallowing makes this kind of break silent.
Environment
c3f3f1cRelated