test(benchmarks): automate Promptfoo setup - #139
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
None.
Summary
npm --prefix benchmarks/codex-mcp run setupentry point for the Codex MCP ablationThis is internal-only benchmark tooling. It does not change a public product interface, storage format, or model format, and existing repositories do not need to be rebuilt.
Validation
npm --prefix benchmarks/codex-mcp run test:setup— 5 Node unit tests passed for version checks, cross-platform paths, environment serialization, index completeness, and native binding selectionnpm --prefix benchmarks/codex-mcp run check— Promptfoo 0.122.2 loaded its Windows libsql binding and reportedConfiguration is valid.npm --prefix desktop run check— TypeScript, ESLint, 47 Vitest tests, and the Vite production build completed under Node 24.20.0npx --yes markdownlint-cli2@0.23.2— completed with no findingsgit diff --check— completed with no whitespace errorsThe full setup was not run because it requires the 1,063,510,782-byte LongVALE archive, model preparation, and a dedicated interactive Codex login. No Codex inference was run. Lychee was not run locally because Docker Desktop was unavailable; the repository documentation workflow remains responsible for that check on the PR.