[Leaderboard Update] Permute EQ - Claude Opus 5 - 90.58% Pass@1

[ericmillsio]

[Leaderboard Update] Permute EQ - Claude Opus 5 - 90.58% Pass@1

This PR updates the Permute EQ submission from #88 as a followup from emails with the benchmark team.

Prompt assignment

• Prompt 1: AG News, BookReview, CRMArenaPro, and Yelp
• Prompt 2: all remaining queries except GITHUB_REPOS Q2
• Revised Prompt 2: all five GITHUB_REPOS Q2 trials, following the clarified query wording

No query mixes prompt versions. CRM Q3, Q7, and Q9 all use the original Prompt 1.

Replacements

• GITHUB_REPOS Q2: runs 0-4
• crmarenapro Q3: runs 0, 1, 2, and 4
• crmarenapro Q7: runs 0 and 1
• crmarenapro Q9: runs 0, 1, and 3
• stockmarket Q3: run 3

BookReview remains unchanged (the benchmark team confirmed its extraction error separately). The other 255 answer and trace pairs are unchanged from #88.

Revised result: 252/270 raw, 0.9058 stratified Pass@1.

Submission file: leaderboard_submissions/permute_eq.json
Traces: permute_eq_traces.zip

[Ruiying-Ma]

Hi @ericmillsio! Thanks for opening the PR.

We can accept the GITHUB_REPOS Q2 rerun, but not the crmarenapro Q3, Q7, and Q9 ones, because replacing only the trials that failed removes the very failures Pass@1 is meant to count.

This puts you at 0.8922 for now. We can put that on the leaderboard if you are okay with it. Alternatively, we are happy to re-audit if you rerun all 5 trials of those three queries under the same frozen setup and send whatever comes out, including the failures.

[ericmillsio]

Okay thank you clarification. Please hold off merging for now, I will be attempting reruns. Thanks!