Evaluate zero-shot TF-IDF and compression on JevBench

[mrmps]

Measures zero-shot TF-IDF/compression hybrids on JevBench's 231 public typed decisions. Every method sees only the current state, instructions, labels, and option descriptions: no labeled training, support examples, learned weights, or calibration.

Method	Accuracy
Character TF-IDF	49.4%
Best observed fixed TF-IDF + Zstd blend	50.6%
Jev 1.13.0, published on the same public IDs	86.6%

The hybrid is the best of 45 fixed blends in an exploratory sweep. Its three-answer gain over character TF-IDF has a paired 95% interval of −3.07 to +5.73 percentage points, so these results do not establish an improvement. Public accuracy is not an official JevBench score.

The harness includes complete prediction/score artifacts, source hashes, option-order checks, and reproduction instructions. The checked-in report covers all 54 methods. Only offline evaluation files change; visual evidence is not applicable, and deployed behavior is unchanged.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}