From 776b14fc0d884bfdd8e3491278e297e5f0f2c417 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Tue, 10 Mar 2026 07:38:24 +0000 Subject: [PATCH] docs: update workspace docs for static auto-materialise and hooks.enabled Update documentation across SKILL.md, configuration.mdx, workspace-pool.mdx, and running-evals.mdx to reflect PR #515 and PR #516 changes: - Rename ephemeral -> temp in workspace mode enum - Rename static_path -> path - Document hooks.enabled option (boolean, default true) - Document static workspace auto-materialisation when path is empty/missing --- .../content/docs/evaluation/running-evals.mdx | 10 ++++++---- .../src/content/docs/guides/workspace-pool.mdx | 18 ++++++++++-------- .../src/content/docs/targets/configuration.mdx | 13 ++++++++----- .../skills/agentv-eval-builder/SKILL.md | 9 ++++++--- 4 files changed, 30 insertions(+), 20 deletions(-) diff --git a/apps/web/src/content/docs/evaluation/running-evals.mdx b/apps/web/src/content/docs/evaluation/running-evals.mdx index e199d392d..11a24638f 100644 --- a/apps/web/src/content/docs/evaluation/running-evals.mdx +++ b/apps/web/src/content/docs/evaluation/running-evals.mdx @@ -101,7 +101,7 @@ The `--otel-file` format writes standard OTLP JSON that can be imported into any Use workspace mode and finish policies instead of multiple conflicting booleans: ```bash -# Mode: pooled | ephemeral | static +# Mode: pooled | temp | static agentv eval evals/my-eval.yaml --workspace-mode pooled # Static mode path @@ -118,17 +118,19 @@ Equivalent eval YAML: ```yaml workspace: - mode: pooled # pooled | ephemeral | static - static_path: null # required when mode=static + mode: pooled # pooled | temp | static + path: null # workspace path for mode=static; auto-materialised when empty/missing hooks: + enabled: true # set false to skip all hooks after_each: reset: fast # none | fast | strict ``` Notes: - Pooling is default for shared workspaces with repos when mode is not specified. -- `mode: static` (or `--workspace-mode static`) requires `static_path` / `--workspace-path`. +- `mode: static` (or `--workspace-mode static`) uses `path` / `--workspace-path`. When the path is empty or missing, the workspace is auto-materialised (template copied + repos cloned). Populated directories are reused as-is. - Static mode is incompatible with `isolation: per_test`. +- `hooks.enabled: false` skips all lifecycle hooks (setup, teardown, reset). - Pool slots are managed separately (`agentv workspace list|clean`). ### Retry Execution Errors diff --git a/apps/web/src/content/docs/guides/workspace-pool.mdx b/apps/web/src/content/docs/guides/workspace-pool.mdx index d7bab9c13..8b06f2b08 100644 --- a/apps/web/src/content/docs/guides/workspace-pool.mdx +++ b/apps/web/src/content/docs/guides/workspace-pool.mdx @@ -40,14 +40,14 @@ Pooling is on by default. To disable it: ### CLI mode ```bash -agentv eval evals/my-eval.yaml --workspace-mode ephemeral +agentv eval evals/my-eval.yaml --workspace-mode temp ``` ### YAML workspace mode ```yaml workspace: - mode: ephemeral + mode: temp repos: - path: ./my-repo source: @@ -55,7 +55,7 @@ workspace: url: https://github.com/org/my-repo.git ``` -`workspace.mode` controls materialization behavior directly (`pooled`, `ephemeral`, or `static`). +`workspace.mode` controls materialization behavior directly (`pooled`, `temp`, or `static`). ## Pool reset mode @@ -169,13 +169,15 @@ This pattern is especially valuable with pooling: a single `workspace.yaml` guar ## Static workspaces (`mode: static`) -For workspaces you manage entirely outside AgentV, use static mode to skip all materialization: +For workspaces you manage outside AgentV, use static mode: ```bash agentv eval evals/my-eval.yaml --workspace-mode static --workspace-path /path/to/my-workspace ``` -This bypasses clone, copy, and pool entirely. AgentV never deletes a user-provided workspace. Lifecycle hooks still execute. This is useful for local development where you already have the repo checked out. +**Auto-materialisation:** When `workspace.path` points to an empty or missing directory, AgentV automatically copies the template and clones repos into it. If the directory already exists and is populated, it is reused as-is. This makes static mode convenient for first-run bootstrap without manual workspace preparation. + +When the directory is already populated, clone, copy, and pool are bypassed entirely. AgentV never deletes a user-provided workspace. Lifecycle hooks still execute (unless `hooks.enabled: false`). This is useful for local development where you already have the repo checked out. **Precedence:** `workspace.mode` / `--workspace-mode` first, then default pooled behavior for shared repo workspaces. @@ -192,13 +194,13 @@ CLI flags `--retain-on-success` / `--retain-on-failure` control temporary eval-r | Mode | Setup cost | Persistent | Build artifacts preserved | Concurrent workers | |------|-----------|-----------|--------------------------|-------------------| | **Pooled** (default) | First run only; reset on reuse | Yes | Yes (`.gitignore`d files) | Yes (slot per worker) | -| **Ephemeral** (`mode: ephemeral`) | Full clone + checkout every run | No | No | Sequential only | -| **Static** (`mode: static`) | None (user-managed) | Yes | User-managed | Sequential only | +| **Temp** (`mode: temp`) | Full clone + checkout every run | No | No | Sequential only | +| **Static** (`mode: static`) | None if populated; auto-materialised if empty/missing | Yes | User-managed | Sequential only | ## When to disable pooling **Pooling is typically the right default.** Consider disabling it when: - You need guaranteed clean-slate isolation between runs - You're debugging workspace setup issues and want fresh clones each time -- You use `mode: static` with a pre-existing directory (pooling is automatically skipped) +- You use `mode: static` with a pre-existing or auto-materialised directory (pooling is automatically skipped) - You need `isolation: per_test` (each test gets its own workspace copy; pooling is automatically skipped) diff --git a/apps/web/src/content/docs/targets/configuration.mdx b/apps/web/src/content/docs/targets/configuration.mdx index 7e75b8537..b86d00d4d 100644 --- a/apps/web/src/content/docs/targets/configuration.mdx +++ b/apps/web/src/content/docs/targets/configuration.mdx @@ -188,8 +188,8 @@ workspace: after_each: reset: fast # none | fast | strict isolation: shared # shared (default) | per_test - mode: pooled # pooled | ephemeral | static - static_path: /tmp/my-ws # required when mode=static + mode: pooled # pooled | temp | static + path: /tmp/my-ws # workspace path for mode=static ``` | Field | Description | @@ -204,10 +204,13 @@ workspace: | `repos[].clone.sparse` | Sparse checkout paths | | `hooks.after_each.reset` | Reset policy after each test: `none`, `fast`, `strict` | | `isolation` | `shared` reuses one workspace; `per_test` creates a fresh copy per test | -| `mode` | Workspace mode: `pooled`, `ephemeral`, `static` | -| `static_path` | Existing workspace path used when `mode=static` | +| `mode` | Workspace mode: `pooled`, `temp`, `static` | +| `path` | Workspace path for `mode=static`. When empty or missing, the workspace is auto-materialised (template copied + repos cloned). Populated directories are reused as-is. | +| `hooks.enabled` | Boolean (default: `true`). Set `false` to skip all lifecycle hooks. | -**Pooling:** `mode: pooled` (or default shared repo mode) reuses pool slots between runs. Use `mode: ephemeral` to disable pooling for fresh clone/checkouts each run. +**Pooling:** `mode: pooled` (or default shared repo mode) reuses pool slots between runs. Use `mode: temp` to disable pooling for fresh clone/checkouts each run. + +**Static auto-materialisation:** When `mode: static` and `path` points to an empty or missing directory, AgentV automatically copies the template and clones repos into it. If the directory already exists and is populated, it is reused as-is. Pool management commands: - `agentv workspace list` — list all pool entries with size and repo info diff --git a/plugins/agentv-dev/skills/agentv-eval-builder/SKILL.md b/plugins/agentv-dev/skills/agentv-eval-builder/SKILL.md index 98864a358..ef8827d55 100644 --- a/plugins/agentv-dev/skills/agentv-eval-builder/SKILL.md +++ b/plugins/agentv-dev/skills/agentv-eval-builder/SKILL.md @@ -252,7 +252,9 @@ workspace: after_each: reset: fast # none | fast | strict isolation: shared # shared | per_test - mode: pooled # pooled | ephemeral | static + mode: pooled # pooled | temp | static + hooks: + enabled: true # set false to skip all hooks ``` - `source.type`: `git` (URL) or `local` (path) @@ -260,8 +262,9 @@ workspace: - `clone.depth`: shallow clone depth - `clone.filter`: partial clone filter (e.g., `blob:none`) - `clone.sparse`: sparse checkout paths array -- `mode`: `pooled` (default for shared repos), `ephemeral`, or `static` -- `static_path`: required when `mode: static` +- `mode`: `pooled` (default for shared repos), `temp`, or `static` +- `path`: workspace path used when `mode: static`; when empty/missing the workspace is auto-materialised (template copied + repos cloned); populated dirs are reused as-is +- `hooks.enabled`: boolean (default `true`); set `false` to skip all lifecycle hooks - Pool reset defaults to `fast` (`git clean -fd`); use `--workspace-clean full` for strict reset (`git clean -fdx`) - Pool entries are managed separately via `agentv workspace list` and `agentv workspace clean`