Vieval

Vitest-style evaluation framework for agents, models, and task pipelines.

vieval keeps eval authoring close to product code while giving you repeatable task discovery, matrix scheduling, live CLI output, JSON artifacts, and report commands.

Why Vieval

Familiar eval files with describeTask, caseOf, casesFromInputs, and expect.
Project, eval, and task matrix layers for model, scenario, rubric, and dataset variants.
Built-in chat-model registration through ChatModels, plus custom project executors for non-chat workloads.
Human-readable terminal output and machine-readable JSON/report artifacts from the same CLI.
Importable runner, scheduler, assertion, config, plugin, and testing entrypoints for advanced integration.

Quick Start

Step 1. Create a config

// vieval.config.ts
import { cwd } from 'node:process'

import { defineConfig, loadEnv, requiredEnvFrom } from 'vieval'
import { chatModelFrom, ChatModels } from 'vieval/plugins/chat-models'

export default defineConfig({
  env: loadEnv('test', cwd(), ''),
  plugins: [
    ChatModels({
      models: [
        chatModelFrom({
          aliases: ['agent-mini', 'judge-mini'],
          apiKey: config => requiredEnvFrom(config.env, {
            name: 'OPENAI_API_KEY',
            type: 'string',
          }),
          inferenceExecutor: 'openai',
          model: 'gpt-4.1-mini',
        }),
      ],
    }),
  ],
  projects: [
    {
      evalMatrix: {
        extend: {
          rubric: ['default'],
        },
      },
      include: ['evals/*.eval.ts'],
      name: 'default',
      root: '.',
      runMatrix: {
        extend: {
          model: ['agent-mini'],
          scenario: ['baseline'],
        },
      },
    },
  ],
})

Step 2. Create an eval task

// evals/smoke.eval.ts
import { caseOf, describeTask, expect } from 'vieval'

describeTask('smoke', () => {
  caseOf('arithmetic-default', (context) => {
    expect(context.task.matrix.run.scenario).toBe('baseline')
    expect(2 + 2).toBe(4)
  }, {
    input: {
      prompt: 'Check simple arithmetic.',
    },
  })
})

Step 3. Run

pnpm -F vieval eval:run -- --config ./vieval.config.ts

The published binary form is:

vieval run --config ./vieval.config.ts

Authoring API

Use describeTask for the common Vitest-like authoring path:

import { caseOf, describeTask, expect } from 'vieval'
import { modelFromRun } from 'vieval/plugins/chat-models'

describeTask('prompt-language-ablation', () => {
  caseOf('resolves matrix axes', async (context) => {
    const selectedModel = modelFromRun(context, { axis: 'model' })
    const language = context.task.matrix.run.promptLanguage
    const scenario = context.task.matrix.run.scenario

    expect(selectedModel.id.length).toBeGreaterThan(0)
    expect(language).toBeDefined()
    expect(scenario).toBeDefined()
  }, {
    input: {
      prompt: 'Summarize the position in one sentence.',
    },
  })
})

Use builder style when loading a batch of inputs:

import { describeTask, expect } from 'vieval'

const arithmeticCases = [
  { input: { a: 1, b: 2, expected: 3 }, name: 'addition-small' },
  { input: { a: 20, b: 22, expected: 42 }, name: 'addition-large' },
]

describeTask('arithmetic-quality', ({ casesFromInputs }) => {
  casesFromInputs('arithmetic-case', arithmeticCases, ({ matrix }) => {
    const result = matrix.inputs.input.a + matrix.inputs.input.b
    expect(result).toBe(matrix.inputs.input.expected)
  })
})

describeEval remains exported as an alias of describeTask, but new examples should prefer describeTask because task/case semantics are the primary runtime model.

Matrix Model

vieval expands matrix scopes in this order:

Project config from vieval.config.*.
Eval definition from defineEval(...).
Task definition from defineTask(...).

Within each scope, matrix layers resolve in this order:

disable
extend
override

Both runMatrix and evalMatrix are supported at project, eval, and task scope. A flat object such as runMatrix: { scenario: ['baseline'] } is normalized to runMatrix.extend; layered form is preferred for new docs and examples.

Each scheduled task receives stable matrix metadata:

task.matrix.run
task.matrix.eval
task.matrix.meta.runRowId
task.matrix.meta.evalRowId
task.matrix.inputs for caseOf(..., { input }) and casesFromInputs(...)

Orchestration Model

vieval separates benchmark management from run reliability:

comparison: cross-project or cross-workspace benchmark. Use it for horizontal evaluation across multiple agent, memory, model, paper, or backend implementations. Methods do not need perfect project/case alignment; compare artifacts report project and case coverage.
workspace: a batch-management boundary for related eval projects. Use it when one benchmark family spans multiple task projects, roots, env settings, or model registrations.
project: one eval task project with discovery rules, model registrations, optional executor, matrix layers, and scoring/reporting behavior.
experiment: run metadata derived from explicit --experiment or, when omitted, stable matrix row metadata. It does not create an extra scheduler layer.
task: one eval definition discovered from files and expanded across inference executors plus run/eval matrix rows.
case: the scoring and evidence source inside a task. context.score(...) contributes normalized score evidence; context.metric(...) emits benchmark metadata for reports.
attempt: a full task rerun used to estimate reliability. With autoAttempt, each full attempt contributes evidence, so a fail-then-pass pair scores as a success rate rather than replacing the earlier failure.
retry: an in-case retry. With autoRetry, a case can recover inside one attempt; a retry pass still counts as that attempt passing.

Config inheritance follows the same outside-in model: top-level defaults apply first, workspace/project entries refine them, and project-local plugins can append or override project-local models/reporters/concurrency without leaking to sibling projects.

Config Example

import { cwd } from 'node:process'

import { defineConfig, loadEnv, requiredEnvFrom } from 'vieval'
import { chatModelFrom, ChatModels } from 'vieval/plugins/chat-models'

export default defineConfig({
  env: loadEnv('test', cwd(), ''),
  plugins: [
    ChatModels({
      models: [
        chatModelFrom({
          aliases: ['agent-mini', 'judge-mini'],
          apiKey: config => requiredEnvFrom(config.env, {
            name: 'OPENAI_API_KEY',
            type: 'string',
          }),
          inferenceExecutor: 'openai',
          model: 'gpt-4.1-mini',
        }),
        chatModelFrom({
          aliases: ['agent-large', 'judge-large'],
          apiKey: config => requiredEnvFrom(config.env, {
            name: 'OPENAI_API_KEY',
            type: 'string',
          }),
          inferenceExecutor: 'openai',
          model: 'gpt-4.1',
        }),
        chatModelFrom({
          aliases: ['agent-openrouter-mini'],
          apiKey: config => requiredEnvFrom(config.env, {
            name: 'OPENROUTER_API_KEY',
            type: 'string',
          }),
          inferenceExecutor: 'openrouter',
          model: 'openai/gpt-4.1-mini',
        }),
      ],
    }),
  ],
  projects: [
    {
      evalMatrix: {
        extend: {
          rubric: ['strict', 'lenient'],
          rubricModel: ['judge-mini', 'judge-large'],
        },
      },
      include: ['evals/*.eval.ts'],
      name: 'chat-evals',
      root: '.',
      runMatrix: {
        extend: {
          model: ['agent-mini', 'agent-large'],
          promptLanguage: ['en', 'zh'],
          scenario: ['baseline', 'stress'],
        },
      },
    },
  ],
})

Custom Executor

If a project provides no executor, vieval run still discovers eval files, schedules tasks, and executes module-defined task callbacks. Provide projects[].executor when a project needs custom execution for ASR, TTS, image, motion, hosted agents, or another domain runtime.

import { defineConfig } from 'vieval'

export default defineConfig({
  projects: [
    {
      async executor(task, context) {
        const model = context.models.find(model =>
          model.id === 'motion-default'
          || model.model === 'motion-default'
          || model.aliases.includes('motion-default'),
        )

        if (model == null) {
          throw new Error('Missing configured model "motion-default".')
        }

        const success = model.model === 'v2' && task.matrix.run.scenario === 'baseline'

        return {
          entryId: task.entry.id,
          id: task.id,
          inferenceExecutorId: task.inferenceExecutor.id,
          matrix: task.matrix,
          scores: [{ kind: 'exact', score: success ? 1 : 0 }],
        }
      },
      include: ['evals/*.eval.ts'],
      inferenceExecutors: [{ id: 'motion-engine' }],
      models: [
        {
          aliases: ['motion-default'],
          id: 'motion-engine:v2',
          inferenceExecutor: 'motion-engine',
          inferenceExecutorId: 'motion-engine',
          model: 'v2',
        },
      ],
      name: 'motion-evals',
      root: '.',
    },
  ],
})

CLI

vieval run [--config <path>] [--project <name>] [--json] [--report-out <path>]
vieval compare [--config <path>] [--comparison <id>] [--output <path>] [--format table|json]
vieval report analyze <report-directory>
vieval report index <report-directory> [--output <path>] [--format table|json|jsonl]
vieval report cases <report-directory> [--where <key=value>] [--group-by <key>] [--format table|json|jsonl]
vieval report compare <left-report-directory> <right-report-directory> [--case-key <key>] [--score-kind <kind>] [--format table|json]

Common workspace commands:

pnpm install
pnpm -F vieval eval:run
pnpm -F vieval eval:run -- --config ./vieval.config.ts
pnpm -F vieval eval:run -- --config ./vieval.config.ts --project chess --project moderation
pnpm -F vieval eval:run -- --json
pnpm -F vieval eval:run -- --report-out .vieval/reports --workspace local --experiment prompt-v2 --attempt attempt-a
pnpm -F vieval exec tsx src/bin/vieval.ts compare --config ./vieval.config.ts --comparison agent-memory
pnpm -F vieval exec tsx src/bin/vieval.ts report analyze .vieval/reports/my-run
pnpm -F vieval eval:run -- --help

Concurrency flags are available on vieval run:

--workspace-concurrency
--project-concurrency
--task-concurrency
--attempt-concurrency
--case-concurrency

Public Entrypoints

vieval: defineConfig, loadEnv, requiredEnvFrom, describeTask, describeEval, caseOf, casesFromInputs, and expect.
vieval/config: lower-level defineEval, defineTask, matrix types, task context types, model definitions, and plugin contracts.
vieval/plugins/chat-models: ChatModels, ChatProviders, chatModelFrom, chatProviderFrom, chatModelMatrix, runtime config helpers, and chat telemetry helpers.
vieval/core/runner: collection, scheduling, task context, cache runtime, scheduler runtime, execution, and aggregation utilities.
vieval/core/assertions: assertion primitives and pipeline helpers.
vieval/core/inference-executors: env helpers and remote provider executors.
vieval/testing/expect-extensions: Vitest expect extensions for testing eval behavior.

Architecture

flowchart LR
  CLI["src/cli/index.ts\n(runTopLevelCli)"] --> RUN["src/cli/eval-run.ts\n(runEvalRunCli)"]
  CLI --> COMPARE["src/cli/compare.ts\n(runCompareCli)"]
  CLI --> REPORT["src/cli/report-*.ts\n(report commands)"]
  RUN --> ORCH["src/cli/run.ts\n(runVievalCli)"]
  ORCH --> CFG["src/cli/config.ts\n(loadVievalCliConfig)"]
  ORCH --> DISC["src/cli/discovery.ts\n(discoverEvalFiles)"]
  ORCH --> MODULES["src/cli/module-runtime.ts\n(load eval modules)"]
  MODULES --> DSL["src/dsl/task.ts\n(describeTask/caseOf/casesFromInputs)"]
  ORCH --> SCHEDULE["src/core/runner/schedule.ts\n(createRunnerSchedule)"]
  ORCH --> EXEC["src/core/runner/run.ts\n(runScheduledTasks)"]
  EXEC --> CTX["src/core/runner/task-context.ts\n(createTaskExecutionContext)"]
  EXEC --> AGG["src/core/runner/aggregate.ts\n(aggregateRunResults)"]
  ORCH --> REPORTERS["src/cli/reporters/*\nlive reporter + Vitest bridge"]
  ORCH --> ARTIFACTS["src/cli/report-artifacts.ts\nJSONL report artifacts"]
  CHAT["src/plugins/chat-models/*\nmodel/provider plugins"] --> CFG
  PROVIDERS["src/core/inference-executors/*\nprovider adapters + env"] --> CTX

Runtime Sequence

sequenceDiagram
  participant U as User
  participant C as src/cli/index.ts
  participant E as src/cli/eval-run.ts
  participant R as src/cli/run.ts
  participant L as src/cli/config.ts
  participant D as src/cli/discovery.ts
  participant M as src/cli/module-runtime.ts
  participant S as src/core/runner/*
  participant T as src/dsl/task.ts
  participant P as src/cli/reporters/*

  U->>C: vieval run --config ...
  C->>E: runEvalRunCli(argv)
  E->>R: runVievalCli(options)
  R->>L: loadVievalCliConfig()
  R->>D: discoverEvalFiles()
  R->>M: loadEvalModulesWithVitestRuntime()
  M->>T: register describeTask definitions
  R->>S: collectEvalEntries() + createRunnerSchedule()
  R->>P: createCliReporter(isTTY)
  R->>P: onRunStart + onTaskQueued
  R->>S: runScheduledTasks(...)
  S->>P: onTaskStart / onTaskEnd
  S->>T: task.run(context)
  T->>P: reporterHooks.onCaseStart / onCaseEnd
  S-->>R: aggregated run results
  R->>P: onRunEnd + dispose
  R-->>E: CliRunOutput
  E->>U: static summary or JSON

Examples In This Repository

Development

pnpm install
pnpm -F vieval test:run
pnpm -F vieval typecheck
pnpm lint

When To Use / Not Use

Use vieval when:

you want evals close to app code with Vitest-like ergonomics;
you need repeatable matrix experiments and stable run metadata;
you want local diagnostics, CI JSON, and report artifacts from one runner;
you need to evaluate product code or custom agent flows without moving them into a hosted eval system.

Do not use vieval when:

you need hosted dataset management, annotation UI, or SaaS observability out of the box;
you only need a one-off script without reusable eval definitions or matrix scheduling.

Acknowledgements

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 75 Commits
.agents/skills		.agents/skills
.github		.github
.vscode		.vscode
.zed		.zed
docs		docs
evals		evals
examples		examples
packages/vieval		packages/vieval
.editorconfig		.editorconfig
.gitignore		.gitignore
.prototools		.prototools
.tool-versions		.tool-versions
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
README.md		README.md
bump.config.ts		bump.config.ts
cspell.config.yaml		cspell.config.yaml
eslint.config.js		eslint.config.js
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
skills-lock.json		skills-lock.json
tsconfig.json		tsconfig.json
turbo.json		turbo.json
vitest.config.ts		vitest.config.ts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

Vieval

Why Vieval

Quick Start

Step 1. Create a config

Step 2. Create an eval task

Step 3. Run

Authoring API

Matrix Model

Orchestration Model

Config Example

Custom Executor

CLI

Public Entrypoints

Architecture

Runtime Sequence

Examples In This Repository

Development

When To Use / Not Use

Acknowledgements

License

About

Uh oh!

Contributors

Uh oh!

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Vieval

Why Vieval

Quick Start

Step 1. Create a config

Step 2. Create an eval task

Step 3. Run

Authoring API

Matrix Model

Orchestration Model

Config Example

Custom Executor

CLI

Public Entrypoints

Architecture

Runtime Sequence

Examples In This Repository

Development

When To Use / Not Use

Acknowledgements

License

About

Resources

Uh oh!

Stars

Watchers

Forks

Contributors

Uh oh!

Languages