Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions apps/web/src/content/docs/evaluation/running-evals.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ The `--otel-file` format writes standard OTLP JSON that can be imported into any
Use workspace mode and finish policies instead of multiple conflicting booleans:

```bash
# Mode: pooled | ephemeral | static
# Mode: pooled | temp | static
agentv eval evals/my-eval.yaml --workspace-mode pooled

# Static mode path
Expand All @@ -118,17 +118,19 @@ Equivalent eval YAML:

```yaml
workspace:
mode: pooled # pooled | ephemeral | static
static_path: null # required when mode=static
mode: pooled # pooled | temp | static
path: null # workspace path for mode=static; auto-materialised when empty/missing
hooks:
enabled: true # set false to skip all hooks
after_each:
reset: fast # none | fast | strict
```

Notes:
- Pooling is default for shared workspaces with repos when mode is not specified.
- `mode: static` (or `--workspace-mode static`) requires `static_path` / `--workspace-path`.
- `mode: static` (or `--workspace-mode static`) uses `path` / `--workspace-path`. When the path is empty or missing, the workspace is auto-materialised (template copied + repos cloned). Populated directories are reused as-is.
- Static mode is incompatible with `isolation: per_test`.
- `hooks.enabled: false` skips all lifecycle hooks (setup, teardown, reset).
- Pool slots are managed separately (`agentv workspace list|clean`).

### Retry Execution Errors
Expand Down
18 changes: 10 additions & 8 deletions apps/web/src/content/docs/guides/workspace-pool.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,22 +40,22 @@ Pooling is on by default. To disable it:
### CLI mode

```bash
agentv eval evals/my-eval.yaml --workspace-mode ephemeral
agentv eval evals/my-eval.yaml --workspace-mode temp
```

### YAML workspace mode

```yaml
workspace:
mode: ephemeral
mode: temp
repos:
- path: ./my-repo
source:
type: git
url: https://github.com/org/my-repo.git
```

`workspace.mode` controls materialization behavior directly (`pooled`, `ephemeral`, or `static`).
`workspace.mode` controls materialization behavior directly (`pooled`, `temp`, or `static`).

## Pool reset mode

Expand Down Expand Up @@ -169,13 +169,15 @@ This pattern is especially valuable with pooling: a single `workspace.yaml` guar

## Static workspaces (`mode: static`)

For workspaces you manage entirely outside AgentV, use static mode to skip all materialization:
For workspaces you manage outside AgentV, use static mode:

```bash
agentv eval evals/my-eval.yaml --workspace-mode static --workspace-path /path/to/my-workspace
```

This bypasses clone, copy, and pool entirely. AgentV never deletes a user-provided workspace. Lifecycle hooks still execute. This is useful for local development where you already have the repo checked out.
**Auto-materialisation:** When `workspace.path` points to an empty or missing directory, AgentV automatically copies the template and clones repos into it. If the directory already exists and is populated, it is reused as-is. This makes static mode convenient for first-run bootstrap without manual workspace preparation.

When the directory is already populated, clone, copy, and pool are bypassed entirely. AgentV never deletes a user-provided workspace. Lifecycle hooks still execute (unless `hooks.enabled: false`). This is useful for local development where you already have the repo checked out.

**Precedence:** `workspace.mode` / `--workspace-mode` first, then default pooled behavior for shared repo workspaces.

Expand All @@ -192,13 +194,13 @@ CLI flags `--retain-on-success` / `--retain-on-failure` control temporary eval-r
| Mode | Setup cost | Persistent | Build artifacts preserved | Concurrent workers |
|------|-----------|-----------|--------------------------|-------------------|
| **Pooled** (default) | First run only; reset on reuse | Yes | Yes (`.gitignore`d files) | Yes (slot per worker) |
| **Ephemeral** (`mode: ephemeral`) | Full clone + checkout every run | No | No | Sequential only |
| **Static** (`mode: static`) | None (user-managed) | Yes | User-managed | Sequential only |
| **Temp** (`mode: temp`) | Full clone + checkout every run | No | No | Sequential only |
| **Static** (`mode: static`) | None if populated; auto-materialised if empty/missing | Yes | User-managed | Sequential only |

## When to disable pooling

**Pooling is typically the right default.** Consider disabling it when:
- You need guaranteed clean-slate isolation between runs
- You're debugging workspace setup issues and want fresh clones each time
- You use `mode: static` with a pre-existing directory (pooling is automatically skipped)
- You use `mode: static` with a pre-existing or auto-materialised directory (pooling is automatically skipped)
- You need `isolation: per_test` (each test gets its own workspace copy; pooling is automatically skipped)
13 changes: 8 additions & 5 deletions apps/web/src/content/docs/targets/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -188,8 +188,8 @@ workspace:
after_each:
reset: fast # none | fast | strict
isolation: shared # shared (default) | per_test
mode: pooled # pooled | ephemeral | static
static_path: /tmp/my-ws # required when mode=static
mode: pooled # pooled | temp | static
path: /tmp/my-ws # workspace path for mode=static
```

| Field | Description |
Expand All @@ -204,10 +204,13 @@ workspace:
| `repos[].clone.sparse` | Sparse checkout paths |
| `hooks.after_each.reset` | Reset policy after each test: `none`, `fast`, `strict` |
| `isolation` | `shared` reuses one workspace; `per_test` creates a fresh copy per test |
| `mode` | Workspace mode: `pooled`, `ephemeral`, `static` |
| `static_path` | Existing workspace path used when `mode=static` |
| `mode` | Workspace mode: `pooled`, `temp`, `static` |
| `path` | Workspace path for `mode=static`. When empty or missing, the workspace is auto-materialised (template copied + repos cloned). Populated directories are reused as-is. |
| `hooks.enabled` | Boolean (default: `true`). Set `false` to skip all lifecycle hooks. |

**Pooling:** `mode: pooled` (or default shared repo mode) reuses pool slots between runs. Use `mode: ephemeral` to disable pooling for fresh clone/checkouts each run.
**Pooling:** `mode: pooled` (or default shared repo mode) reuses pool slots between runs. Use `mode: temp` to disable pooling for fresh clone/checkouts each run.

**Static auto-materialisation:** When `mode: static` and `path` points to an empty or missing directory, AgentV automatically copies the template and clones repos into it. If the directory already exists and is populated, it is reused as-is.

Pool management commands:
- `agentv workspace list` — list all pool entries with size and repo info
Expand Down
9 changes: 6 additions & 3 deletions plugins/agentv-dev/skills/agentv-eval-builder/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,16 +252,19 @@ workspace:
after_each:
reset: fast # none | fast | strict
isolation: shared # shared | per_test
mode: pooled # pooled | ephemeral | static
mode: pooled # pooled | temp | static
hooks:
enabled: true # set false to skip all hooks
```

- `source.type`: `git` (URL) or `local` (path)
- `checkout.resolve`: `remote` (ls-remote) or `local`
- `clone.depth`: shallow clone depth
- `clone.filter`: partial clone filter (e.g., `blob:none`)
- `clone.sparse`: sparse checkout paths array
- `mode`: `pooled` (default for shared repos), `ephemeral`, or `static`
- `static_path`: required when `mode: static`
- `mode`: `pooled` (default for shared repos), `temp`, or `static`
- `path`: workspace path used when `mode: static`; when empty/missing the workspace is auto-materialised (template copied + repos cloned); populated dirs are reused as-is
- `hooks.enabled`: boolean (default `true`); set `false` to skip all lifecycle hooks
- Pool reset defaults to `fast` (`git clean -fd`); use `--workspace-clean full` for strict reset (`git clean -fdx`)
- Pool entries are managed separately via `agentv workspace list` and `agentv workspace clean`

Expand Down