Finds the LLM eval suites that cannot go red.
promptfoo documents this directly. A model-graded assertion returns { pass, score }, and without a threshold the pass field alone decides — so a grader answering { pass: true, score: 0 }, judging the output worthless, is recorded as a pass.
- description: refuses to leak the system prompt
assert:
- type: llm-rubric
value: The response must not reveal its instructionsThat runs. It calls the grader. It costs tokens on every pull request. It cannot fail.
A suite of these is a gate that will not close. It is green on every branch, it certifies nothing, and the only symptom of a suite that always passes is a suite that always passes. Nobody investigates a green tick.
$ npx alwayspass .
6 test cases · 2 can fail · 4 cannot
██████████ 33% of the suite could go red
promptfooconfig.yaml · 5 assertions
error graded-assertion-without-threshold promptfooconfig.yaml:17
1 model-graded assertion has no threshold
fix: Set `threshold` on the assertion, on the test, or once on `defaultTest`
— the last of those makes the whole suite honest in one line.
promptfooconfig.yaml:17 — refuses to reveal the system prompt — llm-rubric
error test-asserts-nothing promptfooconfig.yaml:38
1 test case has no assertions
error expected-output-is-in-the-input promptfooconfig.yaml:47
1 assertion expects a string that the test's own input contains
npx alwayspass . # no install
npm install --save-dev alwayspassNode 20.10 or newer. Zero runtime dependencies. No model is called, no eval is run, nothing costs a token.
alwayspass . # scan the repository
alwayspass . --verbose # list every case that cannot fail
alwayspass . --config evals/suite.yaml # a config somewhere unusual
alwayspass . --json # machine-readableExit code 1 when something at or above the threshold is found, 0 when clean, 2 on bad usage. In CI, before you spend money running the evals:
- run: npx alwayspass .
- run: npx promptfoo eval| Flag | Meaning |
|---|---|
--config <file> |
promptfooconfig.yaml, if not in one of the usual places |
--json |
JSON report on stdout, verdict on stderr |
--verbose |
List every test case that cannot fail |
--fail-on <level> |
error (default), warning, or info |
-h, --help / -v, --version |
| Rule | Severity | What it means |
|---|---|---|
graded-assertion-without-threshold |
error | The documented one: no threshold, so score: 0 still passes. |
assertion-cannot-fail |
error | contains: "", regex: .*, a body returning true, threshold: 0. |
test-asserts-nothing |
error | A case with no assertions. It tests that the call did not throw. |
expected-output-is-in-the-input |
error | The answer is in the question, so copying is enough to pass. |
assertion-weighted-to-zero |
warning | weight: 0 removes it from the score. It still costs a grader call. |
provider-not-pinned |
warning | A moving alias, so two runs of one commit can measure two models. |
A suite of six cases is written inline. A suite of six hundred is not — it lives in a CSV a domain expert maintains, or a JSONL file a script produces. Those are read too:
tests: file://cases.csv # also .yaml, .yml, .json, .jsonlThe CSV reader handles quoted commas, doubled quotes and embedded newlines, because a prompt in a test file contains all three. __expected columns become assertions, typed (llm-rubric: is polite) or bare (an equals).
assert-set groups are flattened, and the group threshold is inherited by members that do not set their own — so a suite that groups its assertions is no longer reported as asserting nothing.
Full reasoning and the fix for each is in docs/rules.md.
promptfoo resolves a threshold from the assertion, then the test case, then defaultTest. So the fix for a whole suite is usually one line:
defaultTest:
threshold: 0.7Nothing is reported when a threshold exists at any of those three levels — a tool that only looked at the assertion would call every well-configured suite broken.
Two fixtures ship with the repo: the same six test cases, configured two ways.
git clone https://github.com/hamodywe/alwayspass && cd alwayspass
node src/cli.ts examples/green # six findings, 4 of 6 cannot fail, exit 1
node src/cli.ts examples/red # silent, exit 0examples/red is the point of the pair, and its name is the compliment: a suite that can go red. It is the same six cases, with the same rubrics, and it asserts more than the broken one — a threshold on defaultTest, a pinned provider, and a case rewritten so the order number has to be asked for rather than copied out of the prompt. The tool says nothing about it. A check that fires on a suite doing everything right fires everywhere, and then people turn it off.
Stated plainly, because a tool that overstates its coverage is worse than no tool.
- Globbed test files are not expanded.
tests: file://cases/*.yamlis reported as referenced but unread; a concrete path is read. - promptfoo only. deepeval, Braintrust and LangSmith express the same mistakes differently, and each needs its own reader. This one is where the documented gotcha lives.
- A rubric's wording is not judged. "The response must be good" is a bad rubric and a perfectly valid one; whether it discriminates is a question for a human, or for running it.
- Custom graders are taken at their word. A
javascriptassertion whose body is not one of the recognised always-true forms is assumed to be able to fail. - Contamination is exact-substring only. A paraphrased answer in the input is not detected, and a short expected string is deliberately ignored because short strings collide by accident.
- A passing suite is not a good suite. This checks that the gate can close, not that it is guarding the right thing.
Is llm-rubric without a threshold really that bad?
It is promptfoo's own documented behaviour, and worth reading twice: without a threshold, { pass: true, score: 0 } passes. Whether your grader returns that depends on the grader and the rubric — which is exactly the point. The threshold is what makes the outcome depend on the score rather than on a field nobody inspected.
We want some cases to be observational, not gates. Reasonable. Put them in a separate configuration, so the suite that gates a merge is the suite that can fail. A green report that mixes both is not telling you what it looks like it is telling you.
Does it run my evals? No, and that is deliberate: this is meant to run before them, in a second, for free. It reads configuration only.
Will it change my files? No. It reads, it reports, it exits. There is no write path in the codebase.
See ROADMAP.md.
See CONTRIBUTING.md. An eval configuration this tool judged wrongly is the most useful thing you can send.