Skip to content

Add Aletheia HumanEval / HumanEval+ and MBPP / MBPP+ official evaluate scores - #27

Open
shaneraphel wants to merge 2 commits into
evalplus:mainfrom
shaneraphel:aletheia-mbpp-plus-20260910
Open

Add Aletheia HumanEval / HumanEval+ and MBPP / MBPP+ official evaluate scores#27
shaneraphel wants to merge 2 commits into
evalplus:mainfrom
shaneraphel:aletheia-mbpp-plus-20260910

Conversation

@shaneraphel

@shaneraphel shaneraphel commented Sep 9, 2026

Copy link
Copy Markdown

Summary

Test plan

  • pip install evalplus==0.3.1 && evalplus.evaluate --dataset humaneval --samples humaneval/samples.jsonl
  • pip install evalplus==0.3.1 && evalplus.evaluate --dataset mbpp --samples samples.jsonl
  • Confirm results.json still parses and the leaderboard page renders

@shaneraphel

Copy link
Copy Markdown
Author

Official evalplus.evaluate 0.3.1 reprint on the same MBPP+ v0.2.0 pin, parallel=2: MBPP 100.0 / MBPP+ 86.8 (378/378 base, 328/378 plus). Entry size remains 0, open-data NONE, prompted true.

@shaneraphel

Copy link
Copy Markdown
Author

Official evalplus.evaluate 0.3.1 reprint on the same MBPP+ v0.2.0 pin, parallel=2: MBPP 100.0 / MBPP+ 87.0 (378/378 base, 329/378 plus). Entry size remains 0, open-data NONE, prompted true.

@shaneraphel

Copy link
Copy Markdown
Author

Public page: https://github.com/shaneraphel/aletheia

Completions (samples.jsonl, 378 rows): https://gist.github.com/shaneraphel/b0fbe4cde5b83185da043c9e0ac00fd0

SHA-256: ba41a115da629ac4bf55213d4e29e4545994e9a014bbee50fc1f4c384dd9f3b1

Co-authored-by: Cursor <cursoragent@cursor.com>
@shaneraphel
shaneraphel force-pushed the aletheia-mbpp-plus-20260910 branch from a3db743 to e731850 Compare September 10, 2026 07:15
Co-authored-by: Cursor <cursoragent@cursor.com>
@shaneraphel shaneraphel changed the title Add Aletheia MBPP / MBPP+ official evaluate scores Add Aletheia HumanEval / HumanEval+ and MBPP / MBPP+ official evaluate scores Sep 10, 2026
@shaneraphel

Copy link
Copy Markdown
Author

HumanEval / HumanEval+ numbers are now filled from the same official evalplus.evaluate 0.3.1 path as the MBPP row.

MBPP / MBPP+ is unchanged: 100.0 / 87.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant