Skip to content

test(benchmarks): automate Promptfoo setup - #139

Merged
tulayha merged 1 commit into
mainfrom
feat/automate-promptfoo-setup
Sep 1, 2026
Merged

test(benchmarks): automate Promptfoo setup#139
tulayha merged 1 commit into
mainfrom
feat/automate-promptfoo-setup

Conversation

@tulayha

@tulayha tulayha commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Related issue

None.

Summary

  • add a single npm --prefix benchmarks/codex-mcp run setup entry point for the Codex MCP ablation
  • automate dependency installation, isolated Codex authentication, pinned LongVALE download and verification, model preparation, indexing, saved environment configuration, and preflight
  • install only the host-specific Promptfoo libsql binding while continuing to omit unrelated optional providers
  • make repeat setup runs reuse prepared state and skip an already complete pilot index
  • replace the manual PowerShell setup sequence in the benchmark guide with the automated workflow

This is internal-only benchmark tooling. It does not change a public product interface, storage format, or model format, and existing repositories do not need to be rebuilt.

Validation

  • npm --prefix benchmarks/codex-mcp run test:setup — 5 Node unit tests passed for version checks, cross-platform paths, environment serialization, index completeness, and native binding selection
  • npm --prefix benchmarks/codex-mcp run check — Promptfoo 0.122.2 loaded its Windows libsql binding and reported Configuration is valid.
  • npm --prefix desktop run check — TypeScript, ESLint, 47 Vitest tests, and the Vite production build completed under Node 24.20.0
  • npx --yes markdownlint-cli2@0.23.2 — completed with no findings
  • git diff --check — completed with no whitespace errors

The full setup was not run because it requires the 1,063,510,782-byte LongVALE archive, model preparation, and a dedicated interactive Codex login. No Codex inference was run. Lychee was not run locally because Docker Desktop was unavailable; the repository documentation workflow remains responsible for that check on the PR.

@tulayha
tulayha merged commit 446e97c into main Sep 1, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant