feat(checkpoint): periodic/milestone checkpoints so stalled runs are recoverable

AGENTS.md

@@ -81,7 +81,7 @@ make validate-examples    # validate all examples
   - `context.py` - `WorkflowContext` manages accumulated agent outputs with three modes: accumulate, last_only, explicit
   - `router.py` - Route evaluation with Jinja2 templates and simpleeval expressions
   - `limits.py` - Safety enforcement (max iterations, timeout)
-  - `checkpoint.py` - Automatic checkpoint saving on failure and resume support. The engine saves a checkpoint inside its main-loop exception handlers; for a dashboard Stop/Kill that *cancels* the engine task from the CLI wrapper (bypassing those handlers), `WorkflowEngine.handle_dashboard_stop(message)` is invoked from `cli/run.py::_execute_with_stop_signal` after the cancelled task is drained. It writes a best-effort checkpoint and emits a single `workflow_failed` (flagged `stopped_by_user: true`, plus `checkpoint_path` or `checkpoint_unavailable_reason`). The `_last_checkpoint_path` guard is a defensive backstop against repeat/direct invocation — the engine's own terminal paths (e.g. pause→Kill `InterruptError`) re-raise out of the engine and are handled separately by the wrapper, so `handle_dashboard_stop` is reached only for genuinely cancelled tasks. Issue #245.
+  - `checkpoint.py` - Checkpoint save/load/list/cleanup + resume support. `save_checkpoint(error=..., trigger=...)` writes a top-level `trigger` typed `CheckpointTrigger = Literal["failure", "periodic"]`; `error=None` (periodic) writes null `failure.error_type`/`message`. No `CHECKPOINT_VERSION` bump — `trigger` is additive and any unknown/missing on-disk value normalizes to `"failure"` on load. `rotate_periodic_checkpoints` / `cleanup_periodic_for_run` both delegate to `_delete_periodic_checkpoints(..., keep_last, action)` (cleanup == rotate with `keep_last=0`), scoped to `trigger == "periodic"` **and** an exact `run_id` match so failure checkpoints and other runs' files are never touched. `find_latest_checkpoint` returns `list_checkpoints(...)[0]` (newest by microsecond `created_at`, not filename) so resume-latest isn't fooled by same-second periodic checkpoints. The engine saves a checkpoint inside its main-loop exception handlers; for a dashboard Stop/Kill that *cancels* the engine task from the CLI wrapper (bypassing those handlers), `WorkflowEngine.handle_dashboard_stop(message)` is invoked from `cli/run.py::_execute_with_stop_signal` after the cancelled task is drained — it writes a best-effort checkpoint and emits a single `workflow_failed` (flagged `stopped_by_user: true`, plus `checkpoint_path` or `checkpoint_unavailable_reason`). `handle_dashboard_stop` is idempotent via a dedicated `_dashboard_stop_handled` flag (not `_last_checkpoint_path`, which periodic checkpoints also set). Issues #244, #245.
 
 - **executor/**: Agent execution
   - `agent.py` - `AgentExecutor` handles prompt rendering, tool resolution, and output validation for single agents
@@ -135,6 +135,7 @@ make validate-examples    # validate all examples
 - **Tool resolution**: `null` = all workflow tools, `[]` = none, `[list]` = subset
 - **Set step typing**: `output_type` defaults to `auto` (safe YAML parse with `_to_json_safe` normalisation — `datetime`/`date`/`time` → ISO 8601, non-string dict keys and other non-JSON-safe values raise `ExecutionError`). Explicit `string`/`number`/`integer`/`boolean`/`list`/`dict` only valid on single `value:`. `WorkflowContext.store` accepts any JSON-safe value (scalars/lists from `set` steps in addition to the dicts produced by LLM / script / gate / parallel-group outputs); `_add_agent_input` returns the scalar verbatim for `step.output` and raises a clear `KeyError` for `step.output.field` shorthand on non-dict outputs.
 - **Reasoning effort**: `runtime.default_reasoning_effort` sets a workflow-wide default; per-agent `reasoning.effort` overrides it. Allowed values: `low`, `medium`, `high`, `xhigh`. Each provider translates the unified value to its native API (Copilot: `reasoning_effort` on the session, validated against the model's `supported_reasoning_efforts`; Claude: extended thinking with budget mapping low=2048, medium=8192, high=16384, xhigh=32768 tokens, with `temperature` coerced to 1.0 and `max_tokens` bumped to fit the budget). See `examples/reasoning-effort.yaml`.
+- **Periodic checkpoints** (`runtime.checkpoint`, issue #244): opt-in `CheckpointConfig` (`every_agent: bool`, `every_seconds: int|None`, `keep_last: int=5`; `is_enabled = every_agent or every_seconds is not None`). Off by default → failure-only behavior preserved. `WorkflowEngine._maybe_save_periodic_checkpoint()` is called once at the **top of `_execute_loop`** (single choke point), where prior outputs are committed and `_current_agent_name` is the step *about to run* — so a periodic checkpoint reuses failure-checkpoint `current_agent` semantics and resume continues forward with no special-casing. Gated via the `_periodic_checkpoints_active` property (**root engine only**, `_subworkflow_depth == 0`, + `is_enabled`) and skips the first iteration (`limits.current_iteration == 0`). The save decision is `_periodic_checkpoint_due(now)` (`every_agent` OR `every_seconds` throttle; first save always fires). `_save_checkpoint_on_failure` and the periodic path share `_write_checkpoint(error, trigger)` (which best-effort-guards provider `get_session_ids()` so it never raises). The periodic save wraps write+emit+rotate; on any failure it calls `_record_periodic_checkpoint_failure()` which emits a **`checkpoint_save_failed`** event (consecutive-failure count; surfaced by `ConsoleEventSubscriber` + JSONL + dashboard) so a recovery-reliant user isn't silently left without checkpoints. After a save the engine calls `rotate_periodic_checkpoints`; at a terminal **non-resumable** outcome (clean completion via `run()`/`resume()`, or an explicit `status: failed` terminate) `_cleanup_run_periodic_checkpoints()` deletes the run's periodic checkpoints (an unexpected failure leaves them in place alongside the failure checkpoint). `conductor checkpoints` shows a `Trigger` column and `—` for periodic rows' error type. See `examples/periodic-checkpoints.yaml` and `docs/workflow-syntax.md` (Periodic Checkpoints section).
 - **Terminate steps** (`type: terminate`): explicit terminal step with `status` (`success` | `failed`), Jinja2 `reason`, and optional `output_template` (a `dict[str, str]` that replaces `workflow.output:` when set; each value is rendered then passed through `_maybe_parse_json` so `"true"` becomes `True`, `"42"` becomes `42`, JSON literals are parsed). Reaching a terminate step ends the workflow immediately (no routes evaluated after). `success` → CLI exit 0, dashboard ✅, `workflow_completed { termination_reason, terminated_by, is_explicit: true, status }`; runs `on_complete` hook. `failed` → CLI exit 1 (with rendered output JSON still printed to stdout for downstream tooling), dashboard ❌, raises `WorkflowTerminated` (subclass of `ExecutionError`), emits `workflow_failed { error_type: "WorkflowTerminated", is_explicit: true, status, output }`, runs `on_error` hook, and **does not** save an on-failure checkpoint (explicit terminations are intentionally non-resumable). Terminate steps cannot have `routes`, `tools`, `output`, `prompt`, `model`, etc.; cannot be used as parallel-group members or as a for_each inline agent (route to one from those groups' `routes:` instead). Inside a sub-workflow, a `status: failed` terminate is downgraded at the parent boundary to `SubworkflowTerminatedError` (also a subclass of `ExecutionError`) preserving the child's rendered `terminated_output` / `terminated_reason` / `terminated_by` as structured attributes — the parent treats it as a normal sub-workflow failure (its own `workflow_failed` does NOT inherit `is_explicit: true`). For more detail see `examples/terminate.yaml`, `docs/workflow-syntax.md` (Terminate Steps section), and `plugins/conductor/skills/conductor/references/authoring.md`.
 - **Structured `runtime.provider` (Copilot custom routing)**: `runtime.provider` accepts either the bare string shorthand (`provider: copilot`) or a structured `ProviderSettings` object that routes the Copilot SDK at OpenAI-compatible / Azure / Anthropic endpoints (Ollama, vLLM, LM Studio, Azure OpenAI, etc.). Object fields: `name` (defaults to `copilot`), `type` (`openai`|`azure`|`anthropic`), `wire_api` (`completions`|`responses`), `base_url`, `api_key`, `bearer_token`, `headers`, `azure.api_version`. `api_key` and `bearer_token` are `SecretStr` (redacted in `model_dump` / dashboard / event logs). The model is frozen after construction. Custom routing activates only when at least one non-`name` field is set in YAML — ambient `OPENAI_*` env vars never divert default routing on their own. Once activated, missing fields fall back from env vars in this order: `base_url` ← `COPILOT_PROVIDER_BASE_URL` → `OPENAI_BASE_URL`; `api_key` ← `COPILOT_PROVIDER_API_KEY` (only — ambient `OPENAI_API_KEY` is intentionally NOT a fallback to avoid credential leaks); `bearer_token` ← `COPILOT_PROVIDER_BEARER_TOKEN`. The schema rejects every non-`name` field when `name != "copilot"` (structured config for other providers is a follow-up). It also rejects anchorless / broken combinations that would silently no-op at the SDK boundary: `wire_api` / `type` / `headers` / `azure` cannot stand alone without `base_url` / `api_key` / `bearer_token`; empty `headers`, empty `SecretStr`, and `azure: {api_version: null}` are rejected. The resolver raises `ProviderError` when custom routing is activated but every resolved field is falsy (e.g. expected env vars all unset). Custom routing applies to both agent execution and dialog turns so all sessions hit the same endpoint. `--provider <name>` CLI override replaces the whole `ProviderSettings` (logs a notice when YAML had structured fields). See `examples/copilot-local-llm.yaml`.
 

docs/workflow-syntax.md

@@ -55,6 +55,12 @@ workflow:
                                       # every provider-backed agent unless it
                                       # declares its own `reasoning.effort`.
                                       # See docs/configuration.md#reasoning-effort.
+
+    checkpoint:                       # Optional: periodic checkpoints (off by default)
+      every_agent: true               # Save after each step boundary (governs alone when true)
+      every_seconds: 300              # Throttle: save at most this often (used only when every_agent is false)
+      keep_last: 5                    # Retain this many periodic checkpoints per run
+
     default_context_tier: default     # Optional: default | long_context (Copilot only)
                                       # Workflow-wide default for the model's
                                       # context-window tier. Inherited by every
@@ -983,6 +989,63 @@ workflow:
 - Includes all agent execution time and overhead
 - `None` (default) means no timeout
 
+### Periodic Checkpoints
+
+By default Conductor writes a checkpoint **only when a workflow fails** with an
+exception. A long run that *stalls* (a provider hang, an MCP deadlock, a network
+blip, a sub-agent that never returns) produces no recoverable state, so
+`conductor resume` has nothing to resume.
+
+Enable **periodic checkpoints** to make stalled or hard-killed runs resumable:
+
+```yaml
+workflow:
+  runtime:
+    checkpoint:
+      every_seconds: 300    # Save at most once every 5 minutes (throttle)
+      keep_last: 5          # Retain this many periodic checkpoints per run (1-100)
+      # every_agent: true   # Alternative: save after EVERY step boundary
+```
+
+- **`every_agent`** (default `false`) — save at every step boundary (after each
+  agent, parallel group, for-each group, gate, script, set, wait, or sub-workflow
+  step). When `true` it governs on its own and `every_seconds` is ignored.
+- **`every_seconds`** (default `null`) — a throttle: save at the first step
+  boundary reached after this many seconds have elapsed since the last
+  checkpoint. The first periodic checkpoint of a run fires at the first
+  boundary; the interval only throttles subsequent saves.
+- Set either trigger (or both — a save fires when **either** is met).
+- **`keep_last`** (default `5`) — older periodic checkpoints for the run are
+  rotated away after each save; **failure checkpoints are never rotated**.
+
+How it works:
+
+- Checkpoints are evaluated at **step boundaries**, where all prior step outputs
+  are already committed. The checkpoint points at the step that was *about to
+  run*, so `conductor resume` continues forward and re-runs only that step.
+- There is no background timer. If a single step runs longer than
+  `every_seconds`, the recovery point is the boundary checkpoint taken **before**
+  that step started — which is exactly what you resume from after killing a
+  stalled run.
+- Periodic checkpoints are written by the **root** workflow only (sub-workflow
+  state is re-run from scratch on resume) and are **deleted automatically when
+  the run reaches a terminal, non-resumable outcome** (clean completion or an
+  explicit `status: failed` terminate). On an unexpected failure they are kept
+  alongside the on-failure checkpoint.
+- If a periodic save itself fails (e.g. the disk fills), the run is not
+  interrupted; the failure is surfaced via a `checkpoint_save_failed` event and
+  a console warning so you know recovery may be unavailable.
+
+Recover a stalled run by killing the process (e.g. `conductor stop` for a
+`--web-bg` run) and then:
+
+```bash
+conductor checkpoints workflow.yaml     # list checkpoints (Trigger column shows periodic/failure)
+conductor resume workflow.yaml          # resume from the latest checkpoint
+```
+
+See `examples/periodic-checkpoints.yaml` for a complete example.
+
 ## Tools
 
 Tools can be configured at workflow or agent level.

examples/periodic-checkpoints.yaml

@@ -0,0 +1,107 @@
+# Periodic / Milestone Checkpoints
+#
+# This example demonstrates opt-in periodic checkpointing (issue #244) so that
+# a long-running workflow that stalls or is hard-killed stays recoverable —
+# without ever raising an exception.
+#
+# By default Conductor only checkpoints on failure. The `runtime.checkpoint`
+# block below saves a resumable checkpoint at each step boundary, so if any
+# stage hangs you can kill the run and pick up where you left off.
+#
+# It shows:
+# - runtime.checkpoint with every_seconds throttling + keep_last rotation
+#   (every_agent is shown as a commented alternative)
+# - a multi-stage pipeline where each stage is an expensive, long-running step
+#
+# Usage:
+#   conductor run examples/periodic-checkpoints.yaml --input topic="vector databases"
+#
+# If the run stalls, kill it, then:
+#   conductor checkpoints examples/periodic-checkpoints.yaml   # inspect (Trigger column)
+#   conductor resume examples/periodic-checkpoints.yaml        # continue from the last step
+
+workflow:
+  name: periodic-checkpoints
+  description: A multi-stage research pipeline with periodic checkpoints
+  version: "1.0.0"
+  entry_point: research
+
+  runtime:
+    provider: copilot
+    checkpoint:
+      # Periodic checkpoints make a stalled or hard-killed run resumable. There
+      # are two triggers, OR-combined — set either one (or both):
+      #
+      #   every_agent: true
+      #     Save after EVERY step boundary. Best when steps are few or each is
+      #     expensive. When true it governs on its own and `every_seconds` below
+      #     is ignored (a save already fires at every boundary).
+      #
+      # `every_seconds` is a throttle: save at the first boundary reached once
+      # this many seconds have elapsed since the last checkpoint. Best for long
+      # runs that mix quick and slow steps, so you don't checkpoint after every
+      # fast step. The first checkpoint of the run is taken at the first boundary;
+      # the interval only throttles subsequent saves.
+      every_seconds: 300
+      # Retain at most this many periodic checkpoints per run. Failure
+      # checkpoints are never rotated.
+      keep_last: 5
+
+  input:
+    topic:
+      type: string
+      required: true
+      description: The topic to research and summarize
+
+  limits:
+    max_iterations: 20
+
+agents:
+  - name: research
+    description: Gather raw findings about the topic (long-running)
+    prompt: |
+      Research the topic below and produce a thorough list of findings,
+      claims, and sources.
+
+      Topic: {{ workflow.input.topic }}
+    output:
+      findings:
+        type: string
+        description: Raw research findings
+    routes:
+      - to: analyze
+
+  - name: analyze
+    description: Analyze the findings and extract key themes
+    prompt: |
+      Analyze the following research findings and extract the key themes,
+      tensions, and open questions.
+
+      Findings:
+      {{ research.output.findings }}
+    output:
+      themes:
+        type: string
+        description: Key themes and analysis
+    routes:
+      - to: summarize
+
+  - name: summarize
+    description: Write a concise summary for the reader
+    prompt: |
+      Write a concise, well-structured summary of the topic for a technical
+      reader, grounded in the analysis below.
+
+      Topic: {{ workflow.input.topic }}
+
+      Analysis:
+      {{ analyze.output.themes }}
+    output:
+      summary:
+        type: string
+        description: Final summary
+    routes:
+      - to: $end
+
+output:
+  summary: "{{ summarize.output.summary }}"

plugins/conductor/skills/conductor/references/authoring.md

@@ -20,6 +20,10 @@ workflow:
     max_agent_iterations: 50     # Max tool-use roundtrips per agent (1-500, optional)
     max_session_seconds: 120     # Wall-clock timeout per agent session (optional)
     default_reasoning_effort: medium  # Workflow-wide reasoning effort: low, medium, high, xhigh (optional)
+    checkpoint:                  # Periodic checkpoints for resumable stalled runs (optional, off by default)
+      every_agent: true          #   save after each step boundary (governs alone when true)
+      every_seconds: 300         #   throttle by elapsed seconds (used only when every_agent is false)
+      keep_last: 5               #   retain this many periodic checkpoints per run (1-100)
 
   input:                         # Define workflow inputs
     param_name:
@@ -62,6 +66,33 @@ workflow:
     # context is loaded at run time instead of being baked into the YAML.
 ```
 
+### Periodic Checkpoints (`runtime.checkpoint`)
+
+Off by default — Conductor otherwise checkpoints only on failure, so a *stalled*
+long run (provider hang, MCP deadlock, sub-agent that never returns) leaves
+nothing to `conductor resume`. Enable periodic checkpoints to make stalled or
+hard-killed runs recoverable:
+
+- `every_agent: true` — save at every step boundary. Governs alone when true
+  (`every_seconds` is then ignored).
+- `every_seconds: N` — throttle: save at the first boundary past N seconds since
+  the last save (set either trigger or both; a save fires when **either** is
+  met). The first save of a run fires at the first boundary; the interval only
+  throttles later saves.
+- `keep_last` (default 5) — rotate older periodic checkpoints for the run;
+  failure checkpoints are never rotated.
+
+Semantics: checkpoints are taken at step boundaries (all prior outputs
+committed) and point at the step *about to run*, so resume continues forward and
+re-runs only that step. No background timer — if a step runs longer than
+`every_seconds`, the recovery point is the boundary checkpoint taken before it
+started. Root workflow only (sub-workflows re-run from scratch on resume), and
+periodic checkpoints are deleted automatically at a terminal non-resumable
+outcome (clean completion or explicit `status: failed` terminate). A failed
+periodic save never interrupts the run — it emits a `checkpoint_save_failed`
+event + console warning. See
+`examples/periodic-checkpoints.yaml`.
+
 ## Agent Definition
 
 ```yaml