Skip to content

bug: enabled schedule shows "Next: 1d ago" — stale next_run_at, cron fires silently skipped with no execution record #1472

Description

@vybe

Summary

On Agent Detail → Schedules, an Active daily schedule (0 4 * * *, Europe/Kiev) displays "→ Next: 1d ago" with 0 runs recorded (7d success ). A "next run" timestamp in the past on an enabled schedule is a contradiction: at least two cron windows passed with no execution row, no skip record, and no advance of the stored next_run_at. A sibling schedule on the same agent fired successfully inside that same window, so the scheduler container was up and its sync loop was running.

Context

agent_schedules.next_run_at is a stored projection, written at creation (src/backend/db/schedules.py:_calculate_next_run_at), at _add_job time (startup + sync), and on the post-fire paths. Several runtime paths let a cron fire pass without advancing the projection or recording anything, which leaves the UI showing a receding "Next: Xd ago":

  1. Autonomy-disabled skipsrc/scheduler/service.py:717-719 returns without creating an execution row and without advancing run times. Fires keep silently skipping every window; next_run_at stays frozen, runs stay 0. (Contrast: the pre-check fire=false path at service.py:773-800 correctly records a skipped execution AND advances run times — the autonomy skip predates that pattern.)
  2. Lock-contention returnservice.py:690-693 returns with no advance and no record.
  3. Failure outcome branchservice.py:867-888: the comment says "update schedule run times" but the branch never calls update_schedule_run_times, so a repeatedly-failing schedule also shows a stale past "Next:" (runs > 0 in that variant).
  4. Sync-loop add-failure permanence — if _add_job throws, the exception is swallowed (service.py:399-400) but _sync_agent_schedules still records the snapshot (service.py:473), so the schedule is never retried — no APScheduler job exists until a container restart, and next_run_at stays at its creation-time value.
  5. No runtime repair — the missed-schedule catch-up (Scheduler fires Monday cron jobs on Tuesday (off-by-one day) #145, _get_missed_schedules) only runs at initialize(); nothing detects or repairs a stale next_run_at while the scheduler is running.

Frontend: src/frontend/src/components/SchedulesPanel.vue:321 renders Next: {{ formatRelativeTime(schedule.next_run_at) }} with no overdue handling, so a past timestamp renders as the nonsensical "Next: 1d ago" instead of an explicit overdue/warning state.

This is the same projection-vs-authority class as #1082 (status-as-projection): next_run_at is a projection of APScheduler's in-memory trigger state, and the write discipline around it is leaky.

Acceptance Criteria

  • Root cause identified for the observed case (enabled daily schedule, 0 runs, next_run_at ~1 day in the past, sibling schedule on the same agent firing normally)
  • Every path where a due cron fire is skipped either advances next_run_at or records an observable skip (execution row / event) — in particular the autonomy-disabled skip at service.py:717
  • Failure outcome branch advances run times (or the projection is advanced at fire time regardless of outcome)
  • _sync_agent_schedules no longer permanently orphans a schedule whose _add_job failed (retry on next sync, or surface the failure)
  • UI renders an explicit overdue state (e.g. "Overdue by 1d") instead of "Next: Xd ago" when next_run_at < now on an enabled schedule
  • Consider a canary invariant: enabled, non-deleted schedule with next_run_at older than now − misfire_grace_time (fits the CANARY-001 catalog)

Technical Notes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions