feat(runtime): unified Automation tool — Codex-style heartbeat + cron scheduling - #558
feat(runtime): unified Automation tool — Codex-style heartbeat + cron scheduling#558hqhq1025 wants to merge 25 commits into
Conversation
… scheduling Replaces the fragmented CronCreate/CronDelete/CronList approach (apache#545) with a single `Automation` tool using a `mode` parameter (create/delete/list/pause/resume) and a `kind` parameter (heartbeat vs cron), following Codex Desktop's pattern. Key design decisions: - One tool, one concept: model only decides heartbeat (continue session) vs cron (fresh session) - Schedule supports: cron 5-field expressions, interval seconds, or one-shot delay - Optional durable persistence (JSON file, atomic writes) for cross-restart survival - 7-day auto-expiry, max_fires cap, consecutive-failure auto-pause (5 strikes) - Scheduler tick every 5s, defers when session busy (max 120s then skip) - Desktop + CLI/TUI both integrated with full persistence and turn-tail injection Addresses PR apache#545 reviewer feedback (Astro-Han): - Not in "unhappy middle": ephemeral heartbeats are honestly session-scoped, durable automations persist to disk. Each kind delivers what its API promises. - Trace is free: injectTurn → sendMessage → AgentRun → full RuntimeEvent trace without new event types or data model changes. - No dependency on apache#544 task/run/trace infrastructure. 3 rounds of adversarial review, 26 bugs found and fixed.
Automated coverage for: heartbeat fires on schedule, durable flag, pause/resume/delete lifecycle, turn-tail list, expiry sweep, max_fires cap, consecutive-failure auto-pause, and cron createFreshRun path.
…ehavior Each test creates a deliberately broken version of the system and asserts the expected failure, proving the integration tests are not vacuous: - injectTurn no-op → heartbeat test catches it - maxFires ignored → cap test catches it - expiresAt unchecked → expiry test catches it - pause no-op → lifecycle test catches it - auto-pause missing → failure test catches it - durable flag dropped → persistence test catches it
- Cron parser: range/step boundary (10-30/5 stops at 30), */10, clean timestamps - State: invalid cron rejection, pruneTerminal, skipFire, terminal guard, listAll, registerAll - Storage: 10 tests for automation-store (CRUD, atomic writes, corrupt/wrong-version handling) - Total: runtime 895 + storage 174 = 1069 automation-related tests passing
1564b76 to
e48d8a5
Compare
There was a problem hiding this comment.
PR #558 review: what's between here and a complete, #544-aligned cronjob
I really like where this is heading. One unified tool with two kinds, the 7-day expiry, max_fires, and consecutive-failure caps, plus the mutation tests, all show real care. To land the cronjob #544 is asking for, here are the gaps I found. I've tried to make each one concrete enough that you or a coding agent can pick it up directly. Symbol anchors are exact; line numbers are approximate.
The shape we're aiming at, from #544: every fire becomes a first-class run (agent-run plus trace) with attribution back to its automation; the scheduler's own activity shows up as events carrying next-schedule state; the definition and event models reuse the existing run/session/task layer instead of a new one; and behaviour is specified for failure, cancellation, and timeout, not only the happy path.
G1 [P1] cron is offered to the model but never wired
The schema in automation-tools.ts exposes kind:"cron" and schedule.type:"cron", and the description promises "create fresh session each run." But apps/desktop/src/main/main.ts calls createMainAutomationWiring without createFreshRun, and packages/cli/src/runtime-bootstrap.ts builds AutomationScheduler without it too. So every cron fire lands on markFailure('Cron execution not configured') and auto-pauses after five.
Fix: on desktop, wire createFreshRun by reusing the background-session path in apps/desktop/src/main/bot-incoming-main.ts:~237 (runtime.createSession() then sendMessage()), so a cron really does spawn a session and a run. The CLI/TUI has no multi-session surface, so let each host derive the tool's kinds from the executor it passes in: give createFreshRun, get cron; otherwise heartbeat only. There's a nice precedent for that at apps/desktop/src/main/task-ledger-wiring.ts:~24 (isTaskLedgerToolsEnabled() ? build : []). While you're there, drop schedule.type:"cron" on hosts without cron so the schema doesn't imply something it can't do.
Done when: creating a cron on desktop spawns a real session and run you can see in the trace, and the CLI schema shows no cron affordance.
G2 [P1] a fire can't be traced back to its automation
injectTurn drops the automation id (_automationId on desktop, and the CLI signature only takes sessionId, prompt). The scheduler calls markSuccess(id) with no runId, so lastRunId stays null. And AgentRunLineage (packages/runtime/src/agent-run.ts:~64) has no triggeredBy slot today, only parent/retry/regenerate/branch.
Fix: have injectTurn and createFreshRun return the runId (or a completion promise), and call markSuccess(id, runId). For the link itself, either add a narrow automationId field to the run header across core/storage/runtime, or index runs from the automation event ledger below. Please don't overload lineage for this.
Done when: you can go run to automation and automation to its past runs, both directions.
G3 [P1] the scheduling itself is invisible
automation-scheduler.ts and automation-state.ts emit nothing; fired, skipped, expired, and paused all live in an in-memory Map plus automations.json. Note that RuntimeEvent (packages/core/src/runtime-event.ts:~253) is bound to sessionId/runId/turnId, so a schedule meta-event, which has no run, won't fit there. Worth keeping in mind before wiring it in.
Fix: add a workspace-level AutomationEvent ledger for created/fired/skipped/paused/expired/failed plus next-schedule state. The run a fire produces still goes through agent-run and trace, referenced from that ledger. Keep automations.json for the definition only (config, like a crontab file), and treat the runtime fields (lastRunId, consecutiveFailures, fireCount, lastError) as projections of the ledger rather than the json's own truth.
Done when: "did the 9am automation run yesterday, did it pass, when's next" is answerable without reading process memory.
G3b [P1] the fire's run reaches the log unlabelled
A fire's turn does reach the event log today: AgentRun.execute() persists inside the generator (agent-run.ts:~457) and both hosts drain the iterator, so the events land. The problem is they land unlabelled. source is hard-coded to 'desktop' in packages/runtime/src/agent-run.ts:~210, and UserMessageInput (packages/core/src/runtime-inputs.ts:31, the sendMessage input) has no origin slot for injectTurn/createFreshRun to pass through. So an automation-triggered run reads exactly like a hand-typed one, and with the dropped automationId from G2 it's effectively an orphan in the log.
Fix: add an origin field to UserMessageInput (it already carries parentRunId/agentId, so an automation origin fits the same shape), have AgentRun read source from the input instead of the constant, and pass it from both injectTurn and createFreshRun. This pairs with the run-header automationId in G2 and gives trace a filterable automation origin.
Done when: trace/inspect can filter runs by automation origin, and an automation-triggered run is never shown as a plain user turn.
G4 [P1] a failed fire can read as success or completed
markFired increments and flips once/maxFires to completed before anything runs. injectTurn gets dispatched and then markSuccess fires right away. Async turn failures are swallowed (.catch(()=>{}) in the CLI, void streamEvents(...) on desktop) and never reach markFailure, and markFailure returns early on a terminal status, so a failed dispatch can still show completed.
Fix: split this into attemptStarted, attemptSucceeded, and attemptFailed; decide the outcome only after the stream finishes; commit terminal completion only on a real success.
Done when: a stream reject, a once reject, and a maxFires=1 reject all end as failed or paused, never completed or success.
G5 [P1] the lifecycle isn't fully closed
In the CLI, cli.ts sets process.exitCode but never calls process.exit, and the scheduler's 5s timer is neither unref'd nor disposed, so the process stays alive and keeps ticking, injecting into a session that's already stopped. Desktop disposes on before-quit, which is good, but session archive/remove never calls removeAllForSession, so non-durable heartbeats leak (only the tests call it today).
Fix: wrap the CLI TUI in try/finally { scheduler.dispose() }, and clear a session's heartbeats on close/archive in desktop. This is also the right moment to define durable resume: what should happen to an in-flight fire after a restart.
Done when: CLI /exit exits cleanly with no further ticks, and a closed desktop session stops firing its heartbeats.
G6 [P2] permission and safety edges
A few things worth a look here. permissionRequired:false means the model can schedule future auto-executing work, including durable and recurring, with no confirmation. canFire only blocks running/blocked, so it misses waiting_for_user/review/done and never asks the runtime whether a run is already active. automation-store returns empty on a corrupt read and a later sync overwrites the file, which is risky once it's a definition source, so it should fail closed or quarantine instead. And two processes on one workspace will start duplicate schedulers, so durable work will eventually want a leader lock or single-instance guard.
G7 [P3] the hand-rolled cron parser
The ~366-day search window rejects valid sparse annual crons like 0 0 29 2 *; day-of-month plus day-of-week is treated as AND where Vixie cron uses OR; and there's no timezone story. Good candidate to pin down in the RFC, or to hand to a vetted parser.
Suggested gating
G1 through G5 are the must-fixes to call this a working cronjob (or split any of them into owned follow-ups with issue links). G6 is a should-fix, and G7 can ride along with the RFC. Since G2 and G3 define a shared data model (the automation-to-run field and the AutomationEvent ledger, plus how cross-session ownership works), it's probably worth nailing that schema first, in this PR or a short RFC. #544 already lists "cronjob/scheduled-run RFC built on task/run trace," and settling it early keeps parallel coding-agent work from diverging.
Happy to pair on the G2/G3 data model before that parallel work kicks off, if it helps. Thanks for pushing this one forward.
…tive 6)
Adds /goal-style autonomous execution: the agent works toward a durable
objective across turns without per-step approval, stopping when an external
evaluator judges the condition met/impossible or a safety cap trips.
Design (CC evaluator + Codex lifecycle):
- External evaluator (CC-style): a cheap-model judge runs after each turn and
returns {met, impossible, progress, waiting}. Keeping the judge external
prevents the working model from rationalizing a premature "done" (Codex's
documented failure mode).
- Evaluate-FIRST ordering: a goal genuinely completed on its final permitted
turn is detected as achieved, not misreported as a cap failure.
- Lifecycle (Codex-inspired): active → achieved/impossible/cleared/paused/
stalled/budget_limited/max_iterations. GoalSet/Clear/Status/Pause/Resume tools.
- Safety caps: block cap (8 consecutive no-progress turns → stalled), token
budget, max iterations (50). Evaluator timeout (30s) + parse failures are
NEUTRAL — a transient/garbled evaluator cannot defeat stall detection.
- Trace is free: continuation goes through runtime.sendMessage → AgentRun →
full RuntimeEvent trace, no new event types.
- Abort halts the loop (desktop turnAborted + CLI !closed guard + canContinue
rejects 'aborted'). Re-entrancy guarded per session.
Integration: desktop + CLI/TUI both wired (tools, turn-tail status injection,
continuation at the turn boundary), session cleanup on archive/remove.
The waiting→heartbeat automation bridge was explored and removed: coupling two
independent lifecycles created a maxFires zombie. A 'waiting' evaluation is now
neutral (no stall) + normal re-check, bounded by maxIterations.
5 rounds of adversarial workflow review, 20 confirmed issues found and fixed
until a clean pass. runtime + desktop test suites green (goal-specific: ~70 tests).
…m (PR apache#558 review G1-G5) Addresses Astro-Han's PR review — the must-fix functional gaps that stopped cron from working: - G1: cron now actually runs. Desktop wires createFreshRun (createSession in explore mode + sendMessage), so a cron fire spawns a real session + run labelled `automation`/`cron`. CLI has no multi-session surface, so cron is gated off there (cronEnabled derives from the executor; the tool advertises heartbeat only and rejects the cron kind at the schema). - G4: outcome is decided AFTER the run's stream finishes. Split the fire into attemptStarted / attemptSucceeded / attemptFailed; terminal completion (once / maxFires) commits only on a real success. injectTurn/createFreshRun now return a Promise<AutomationFireResult>; a rejected or ok:false run is recorded as a failure, never a success. A one-shot failure pauses (visible, not a zombie). - G2: fires carry their runId — markSuccess(id, runId) sets lastRunId. - G3b: automation-triggered runs are labelled. New TurnOrigin on UserMessageInput threads to AgentRunHeader.automationId, so trace can tell an automation run from a hand-typed one. - G5: CLI wraps the TUI in try/finally { scheduler.dispose() } so its timer does not keep the process alive; desktop already clears session heartbeats on archive/remove. - G6 (partial): canFire only fires into a genuinely idle session. Deferred (per reviewer's own sequencing — settle the data model first): G3 AutomationEvent ledger, G7 cron-parser edge cases (sparse annual crons, dom+dow OR semantics, timezone). Will follow up as an RFC/owned issue. Tests updated for the attempt* lifecycle; new cron integration + gating tests.
# Conflicts: # apps/desktop/src/main/main.ts # packages/cli/src/cli.ts # packages/cli/src/runtime-bootstrap.ts
…ropic API)
Caught by a headless end-to-end run against a real Anthropic model: the unified
Automation tool used a discriminatedUnion, which serializes to JSON Schema as
{ anyOf: [...] } with NO top-level "type". Anthropic rejects tool definitions
whose input_schema.type is not "object" ("tools.0.custom.input_schema.type:
Field required", HTTP 500) — and since all tools are sent every turn, this broke
EVERY turn in any session that had the Automation tool registered (the model's
reply came back empty and the session wedged in 'blocked').
Fix: flatten to a single top-level z.object with `mode` (enum) and per-mode
fields optional, validated in impl(). The `kind` enum still gates cron off on
hosts without a fresh-run executor. Nested `schedule` union stays (a union under
a property is fine; only the top level must be an object).
Verified: headless cron run now gets a real LLM reply (assistant="CRON_OK") and
the fresh session settles 'active' instead of 'blocked'.
…viewer G1) createMakaCliRuntimeContext now accepts an optional automationCreateFreshRun. When a host provides it, the Automation tool advertises the cron kind and cron fires spawn a fresh session + run through that executor; omitted (the default CLI, no multi-session surface) means heartbeat only. This matches the reviewer's G1 guidance — a host derives cron support from the executor it passes in. Verified end-to-end through the REAL Maka agent chain (runtime.sendMessage → AgentRun → AiSdkBackend + real Maka system prompt) with natural user phrasing against a real LLM: "每20秒检查系统状态" → heartbeat/interval/20; "工作日9点日报" → cron 0 9 * * 1-5; "5分钟后提醒" → once/300; "长期保留重启别丢" → durable; pause/resume/delete by natural reference (model lists then acts); GoalSet with max_iterations from "最多5轮" and pause. 9/9.
Update — G1–G5 wired, and verified end-to-end against a real modelThanks for the detailed review. I've addressed the must-fix functional gaps and, importantly, tested the whole thing through the real Maka agent chain ( What's fixed
A real bug the end-to-end run caughtThe unified tool originally used a End-to-end results (real chain, natural phrasing, real LLM)Creation + semantics — the model translates intent to the right tool call:
Actual firing (not just creation):
One nice observation: when I first gave a shell-dependent goal in an explore-mode session, the model recognized it couldn't run tests and cleared the goal itself rather than spinning — correct self-management. Deferred (per your sequencing)The bigger data-model items I've left as follow-ups so we can settle the schema first, as you suggested:
Does splitting G3/G6/G7 into an owned follow-up (RFC-first for G3) sound right to you? |
…mezone doc (PR apache#558 G7) - Sparse annual crons: search window 366d → ~8y (MAX_SEARCH_MINUTES). The max gap between Feb 29ths is 8 years, not 4 — a century year not divisible by 400 (2100) is skipped, so 2096→2104. 8y guarantees every satisfiable expression resolves while staying bounded. "0 0 29 2 *" now resolves; "0 0 30 2 *" still returns null after a bounded search (no infinite loop). - day-of-month + day-of-week semantics: when BOTH fields are restricted, a day matches if it satisfies EITHER (Vixie OR), not both. "0 0 13 * 5" now means "the 13th OR any Friday", not "Friday the 13th". When one field is *, AND applies (the * is a no-op). dom-only and dow-only unchanged. - timezone: documented contract (host local time via Date local getters, incl. DST behavior); per-automation IANA zones out of scope for this pass (would ripple through the schedule type and every caller). 8 new tests (Feb 29 resolves, Feb 30 null, dom+dow OR both directions, dom-only, dow-only, regressions for */5, 0 9 * * 1-5, 10-30/5). runtime suite green (pre-existing flaky shell tests aside).
…rectness (self-review)
Adversarial review of the done cronjob work surfaced 5 real bugs; all fixed:
- P1 concurrent cron re-fire: the scheduler had no in-flight guard, and canFire
gates the automation's CREATOR session — which stays idle for cron (the run
happens in a spawned session), so a cron whose run outlasts its cadence
re-fired every tick (duplicate sessions, maxFires blown, out-of-order counter
corruption). Added a per-automation in-flight Set: skip while a fire's run is
executing; add before dispatch, clear in both .then and .catch (and on dispose).
- P2 maxFires not enforced on failure: fireCount++ is unconditional in
attemptStarted but the cap only lived in attemptSucceeded, so a failing
recurring automation fired up to the consecutive-failure cap (5), and fireCount
could exceed maxFires ("Fires: 5/2"). maxFires is now a hard cap on ATTEMPTS —
attemptStarted nulls nextFireAt once fireCount reaches maxFires.
- P2 desktop fires never recorded lastRunId: streamEvents returns {turnId,...}
but the scheduler reads result.runId. Desktop injectTurn/createFreshRun now map
turnId → runId so attemptSucceeded sets lastRunId.
- P2 cron sessions accumulated unbounded: each cron fire spawned a fresh session
forever. createFreshRun now archives the fresh session after its run finalizes
(run/trace preserved, active list not flooded).
- P2 dow=7 never matched: cron allows 0 or 7 for Sunday but Date.getDay() is 0-6.
Added the Sunday 7-alias so "7", "5-7", "0,7" fire on Sundays.
New tests: in-flight guard (slow cron doesn't re-fire concurrently), maxFires
bounds attempts even when every run fails, dow=7 Sunday matching. Runtime +
desktop suites green (pre-existing flaky shell tests aside).
…ew round 2) A maxFires-exhausted (or one-shot that already fired) automation only reaches 'paused' via the attemptFailed path, which leaves nextFireAt=null. resume() previously re-armed it unconditionally, so the next tick bumped fireCount past maxFires (or re-fired the 'once') — spawning a real extra run beyond the declared hard cap. resume() now refuses when the fire budget is spent, and the Automation tool reports the exhausted budget instead of a misleading 'not paused'. Adds two regression tests.
|
Thanks for the fast turnaround, and for testing through the real agent chain instead of a mock harness. That end-to-end pass earned its keep: the I re-verified each fix against the code at the current head. G1 through G5 all hold up.
And yes to your sequencing question: G3 as an RFC-first follow-up sounds right, same for the leader lock and corrupt-store hardening under G6. To be clear up front: none of what follows blocks the merge from my side. Two items are structural, though, and settling them now will be much cheaper than settling them after. 1. Suggest splitting the goal system into its own PR, stacked on this oneThe goal work (goal-state / goal-tools / goal-evaluator / goal-continuation / goal-wiring, roughly 1,200 lines) came in after the review round and is not in the PR description yet. I think it is a strong feature that stands on its own, and an autonomous continuation loop is exactly the kind of code that deserves a focused review rather than riding along in a scheduling PR. Stacking on this branch is fine by me, since it shares the streamEvents seam. Splitting also takes most of my remaining findings with it, because they live in the goal code:
None of these look hard. They just deserve their own review cycle instead of hiding in this diff. 2. Cron needs to own its cwd and survive its creatorTwo related bugs with one root cause: the cron definition does not persist enough about where it came from.
Minimal fix: persist the owner cwd in Smaller items, follow-ups are fine
Overall this landed in very good shape. The state-machine rework came out cleaner than what I sketched in the review, and the live-model verification setup is something I hope we keep using. Happy to review the stacked goal PR whenever it is up. |
…tart Persistence infra (FileAutomationStore + loadDurableAutomations + sync-on- mutation + scheduler.start) was fully wired in both desktop and cli, but durability was opt-in via a `durable` flag defaulting to false for BOTH kinds. A cron is a standalone scheduled task (fresh session each run) — it is pointless if it dies on restart, yet it only persisted when the model happened to pass durable:true. create() now defaults durable by kind: cron=true, heartbeat=false (bound to its session), with an explicit flag always winning. Updates the tool schema description and adds create-default tests.
Goal is independent of Automation (the heartbeat bridge was removed in the self-review pass), so it moves to its own stacked PR. Removes the 9 goal-only files plus every goal wiring point — index.ts exports, cli runtime-bootstrap (GoalManager/buildGoalTools/goalContinuationDeps), cli.ts + pi-tui-runner onTurnComplete hook, cli/desktop turn-tail goal fragments, and desktop main goalWiring (tools, session-lifecycle removal, turn-boundary continuation, quit dispose). Automation is untouched. Goal is re-added verbatim on the stacked branch via the inverse of this commit. Verified: runtime automation 87/0, cli 115/0, desktop main typecheck clean, zero goal/automation-named test failures.
…able across sessions Persisted crons reload from disk under their original sessionId, but list / pause / resume / delete were session-scoped, so after a restart a fresh session could not see or manage them — persistence without query. Durable automations are now app-global: listVisibleForSession surfaces this session's own plus every durable one, and pause/resume/delete accept a durable target from any session. Non-durable heartbeats stay session-private; the per-session create cap still counts only session-owned automations. Adds cross-session unit tests and a real desktop e2e (createMainAutomationWiring + FileAutomationStore on temp disk): create durable cron -> persist -> restart -> fresh session lists/manages/deletes it, deletion re-persisted. Verified end-to-end through the real Maka chain (runtime.sendMessage + real LLM): natural phrasing creates a durable cron that a brand-new session lists after restart.
|
Both follow-ups from the review are up: Goal split → #625 (stacked, draft — review after this merges). This PR is now Automation-only; its net diff vs Persistence + query fixed (
Verified end-to-end through the real Maka chain (
Plus a committed desktop e2e (real |
Astro-Han
left a comment
There was a problem hiding this comment.
The two structural items from my last pass are both resolved: the PR is Automation-only now (goal moved to #625), and app-global durable automations close the "persisted but unqueryable" gap. Cron defaulting to durable is the right call, and the resume budget guard plus the cross-session e2e are good self-review catches.
Approving as promised. The remaining known items stay follow-up material; to keep them from evaporating, let's get issues on:
- cron resolves the project root at fire time, so it runs in whatever project is current (persist the owner cwd in the definition)
- the scheduler still gates cron fires on the creator session, so a deleted creator leaves a durable cron that reloads and is manageable but never fires
- durable load is fire-and-forget while sync replaces the whole file (needs a load barrier)
- resume() re-checks the fire budget but not the 5-active-heartbeat cap
- lastRunId records the turnId rather than the runId
- CLI canFire still admits waiting_for_user/review, and durable/recurring creation could ask for confirmation (G6)
Happy to file these if you'd rather focus on #625. Nice work landing this one.
… lifecycle hardening (adversarial review) Root cause of 3 P1s: canFire gated EVERY automation on its creator session's state, but a cron spawns a FRESH session — so archiving/deleting the creating conversation (or losing it across a restart) permanently stopped a durable cron from ever firing, then it silently expired. canFire is now kind-aware (receives the automation): cron is gated only on the global privacy (incognito) check; heartbeat still requires its own session to exist and be idle. The desktop gate is extracted to a pure, unit-tested evaluateAutomationCanFire. Also from the review: - heartbeat is now ALWAYS session-bound (durable is a cron-only concept) — a durable heartbeat was a post-restart zombie and evaded persistence on removeAllForSession. - registerAll heals an interrupted fire (active + nextFireAt=null, budget not spent → re-armed) instead of leaving a silent zombie until expiry. - resume() resets consecutiveFailures/lastError so a resumed automation isn't re-paused after a single fresh failure. - CLI canFire gates incognito and is kind-aware; CLI durable-sync no longer swallows disk-write errors silently. - desktop canFire drops the dead waiting_for_user branch and no longer throws when a heartbeat's session file is gone. Tests: runtime automation 96/0, cli 118/0, desktop canfire 9/0 + persistence e2e. New: evaluateAutomationCanFire gate, registerAll recovery, resume streak.
…d; heartbeat + incognito gated Ties the P1 fix through the real AutomationManager + AutomationScheduler + the kind-aware evaluateAutomationCanFire gate (injectable timers, no Electron): a durable cron still fires when its creating conversation is archived, while a heartbeat in that same archived session does not, and incognito blocks both.
…ecovery, no once drift - P1 regression: a heartbeat-only host (CLI) sharing the desktop workspace store paused the desktop's durable crons. The cron-without-executor branch called attemptFailed (never advancing nextFireAt) → a tight failure loop that persisted paused state to the shared automations.json. The scheduler now SILENTLY IGNORES a cron when no createFreshRun is configured — no fail, pause, advance, or emit — leaving shared durable state untouched for a host that can run it. - registerAll recovery was inert (it excluded the only naturally-reachable interrupted state). It now settles an interrupted spent-budget fire (once fired / at maxFires, active + nextFireAt=null) to 'completed' (at-most-once, no re-run), and re-arms only a corrupt recurring null. - skipFire on a one-shot settled it via computeNextFire, re-adding the full delay → drift + silent loss under sustained defer (busy/incognito). It now settles a skipped once to 'expired' with a reason instead of drifting. - The kind-aware fire gate moved to @maka/runtime (evaluateAutomationCanFire + HEARTBEAT_IDLE_STATUSES) so desktop and CLI share ONE idle-status definition; the CLI no longer fires a heartbeat into 'waiting_for_user'/'review' sessions. Tests: runtime automation 98/0, cli 118/0, desktop automation 14/0. New: skipFire once-terminal + recurring-advance, registerAll settle-to-completed.
…; interrupted fire records its unknown outcome Two P3s from round-3 (converged from round-2's P1+7): - The eager expiry sweep mutated+persisted crons even on a host without a cron executor, bypassing the 'leave crons untouched' invariant that attemptFire enforces — on a shared workspace a stale heartbeat-only CLI could expire and drop the desktop's cron from automations.json. The sweep now skips crons when createFreshRun is absent, mirroring attemptFire. - registerAll settled an interrupted (crash mid-run) fire to a clean 'completed' with no error, indistinguishable from a real success even though the run's outcome was never committed. It now records lastError='Interrupted on restart ... not re-run.' so the unknown outcome is surfaced (no silent unknown state). Tests: runtime automation 99/0, cli 137/0, desktop automation 14/0. New: sweep-skips-cron-on-cron-disabled-host + settle-records-uncertainty.
… automations (round-4 P1) The CLI shares the desktop's workspace by design (resolveMakaWorkspaceRoot reconstructs the Electron userData path), so its automations.json IS the desktop's. store.sync() is a full-file overwrite. Two P1 data-loss paths: - The heartbeat-only CLI has NO durable automations of its own (heartbeats are never durable), yet syncAutomations wrote its empty/stale durable list over the shared file, ERASING the desktop's crons. - loadDurableAutomations + registerAll adopted+reconciled crons the CLI can't run; the round-3 settle-to-completed then dropped them on the next sync. Root-cause fix: durable persistence is now gated on cron capability. A host without createFreshRun neither loads nor writes the durable store — it leaves that state entirely to the host that owns it. Applied symmetrically in the CLI (runtime-bootstrap) and desktop (automation-wiring). Two cron-enabled hosts sharing a store remains the separate, deferred leader-lock (G6). Tests: runtime automation 99/0, cli 137/0, desktop automation 21/0. New e2e: a cron-disabled host boots on a shared workspace, does heartbeat activity, and the owner's durable cron stays intact on disk.
…ates in O(1) (round-5) Two pre-existing defects surfaced by round-5 (the round-4 clobber class was confirmed fully closed): - Store loadAll() masked a corrupt/unreadable automations.json as an empty store, so a subsequent full-overwrite sync would silently and permanently erase real durable crons (a transient EMFILE/EBUSY or a version-mismatch at startup was enough). loadAll now distinguishes ENOENT (legitimately empty) from a present-but-unreadable file, which it FAILS LOUD on. Both hosts catch that on load and DISABLE persistence (durableStoreReadable=false) so a later mutation can never overwrite data they failed to read. - computeNextCronFire scanned up to ~8 years (4.2M iterations) synchronously in the Electron main process for a schema-valid-but-unsatisfiable expression — a ~1s freeze easily triggered by common LLM output like '0 9 * * MON' (named tokens were unsupported) or an impossible date like '0 0 30 2 *'. It now normalizes+validates in O(1) first: translates named day/month tokens (MON-SUN, JAN-DEC), rejects out-of-range fields, and fast-fails impossible calendar dates (respecting Vixie dom/dow OR-semantics), before any scan. Tests: runtime automation 105/0 (+6 cron validation), cli 137/0, desktop automation 21/0, storage 10/0. Store corrupt/version tests now assert fail-loud.
…k re-fire storm (round-6) computeNextCronFire minute-aligned the scan start with Date.setSeconds(0,0), which round-trips the instant through local wall-clock. During a DST fall-back (the repeated local hour), V8 re-encodes the ambiguous time to the earlier offset, shifting the start ~59 min BEFORE fromTime — so the scan returned a candidate <= fromTime, violating the strictly-after contract. attemptStarted then re-armed nextFireAt to a past time, and checkAndFire re-fired every tick for the whole repeated hour: one daily cron became a storm of duplicate fresh sessions + LLM runs (annual, per DST zone). Now the start is computed in epoch arithmetic (fromTime - fromTime%60000 + 60000), which is offset-safe; candidate wall-clock fields are still read with local getters, so 'N am local' semantics are unchanged and results are byte-identical for all non-DST expressions. Empirically verified (TZ=America/New_York, '30 1 * * *' at 2026-11-01T06:30Z): was returning an equal/past time, now strictly after. Regression test runs the built module in a child process with TZ set. Tests: runtime automation 106/0.
Astro-Han
left a comment
There was a problem hiding this comment.
Approved. I rechecked the current head and the scheduler startup issue is already handled: desktop starts the automation scheduler during background startup and disposes it before quit.
I’d still track a few follow-ups:
- P2: Cron automations use the project root at fire time, not the project root from creation time. If the user creates a cron in project A and later switches to project B, the fresh run can land in B. Persisting the creation cwd/project root on the automation definition would make the run target stable.
- P2:
AutomationispermissionRequired: false, including create/resume/delete for recurring or durable background jobs. Since this can create future work without another user action, I’d prefer mutating modes to go through permission, or split read-only list/status from mutating operations. - P3: Durable automation load is still fire-and-forget. A create during startup can sync the current in-memory manager before the old durable store has finished loading. A small load barrier would close that race.
- P3: The active heartbeat cap can be bypassed with pause, create, then resume. Rechecking the heartbeat cap on resume would keep the limit honest.
Summary
Replaces the fragmented CronCreate/CronDelete/CronList approach (#545) with a single unified
Automationtool, following Codex Desktop's design pattern: one tool, mode-based operations, two kinds (heartbeat vs cron).Supersedes #545 — addresses all of @Astro-Han's review concerns about the "unhappy middle" between ephemeral and durable.
Design
Why one tool instead of three?
CC uses 3+ separate tools (ScheduleWakeup, CronCreate/Delete/List, /schedule Routines). Codex uses one
automation_updatetool with amodeparameter. One tool = lower model cognitive load, clearer mental model, simpler schema.Two kinds
heartbeatcronTrace is free
Every automation fire goes through the standard
sendMessagepipeline, which already creates a fullAgentRunwithRuntimeEventtrace, cost tracking, and tool call recording. No new event types or data model changes needed.Persistence
durable: false(default): session-scoped, dies when session closesdurable: true: persists toworkspace/automations.json, restores on app restartonStateChangecallbackSafety
max_firescap (optional limit on total fires)Files
New (runtime):
automation-state.ts— AutomationManager + cron parser (250 lines)automation-scheduler.ts— tick loop with idle gate + error isolation + onStateChange (150 lines)automation-tools.ts— single Automation tool with discriminatedUnion schema (210 lines)automation.test.ts— 36 state management + cron parser testsautomation-scheduler.test.ts— 12 scheduler testsautomation-integration.test.ts— 9 end-to-end integration tests (covering full test plan)automation-mutation-verify.test.ts— 6 mutation verification tests (proving tests catch broken behavior)New (storage):
automation-store.ts— generic JSON persistence with atomic writes (95 lines)automation-store.test.ts— 10 persistence tests (CRUD, concurrent, corrupt handling)New (desktop):
automation-wiring.ts— manager + scheduler + store + loadDurable + syncOnChangeModified:
packages/runtime/src/index.ts— exportspackages/storage/src/index.ts— exportsapps/desktop/src/main/main.ts— tool registration + scheduler lifecycle + durable loadpackages/cli/src/runtime-bootstrap.ts— CLI tool + scheduler + store + durable syncpackages/cli/src/cli-system-prompt.ts— turn-tail shows active automationsscripts/check-console.mjs— allow automation warn sitesAddressing #545 review feedback
durableparam: false = honestly ephemeral, true = persists to disk with sync-on-every-mutation[Automation: name]\n\nprompt— visible in transcriptQuality
Test coverage