ReviewScore: Misinformed Peer Review Detection with Large Language Models

Updates

Jul 20, 2026: The initial version of the code has been released.
Sep 25, 2025: Our paper is arxived!

Paper

Official source code for "ReviewScore: Misinformed Peer Review Detection with Large Language Models" [1].

TL;DR: We introduce a new evaluation metric of peer review quality, focusing on detecting misinformed review points.

Contents

• data/iclr/papers_iclr.json The paper corpus.
• data/iclr/annotation_iclr.json Human-annotated review points, final labels, and mapped author responses.
• run_eval.py Main script for LLM-based ReviewScore inference and human-model agreement evaluation.
• util.py Shared utilities used by the evaluation script.
• prompts/eval/ Prompt templates for standard ReviewScore evaluation.
• prompts/eval_author_response/ Prompt templates for the author-response variant.
• outputs/outputs_iclr/ Released standard-model evaluation outputs.
• outputs/outputs_iclr_author_response/ Released author-response evaluation outputs.

Annotation Format

data/iclr/annotation_iclr.json is organized as a list of paper-level records. Each paper contains metadata and then groups its annotated review points by reviewer:

[
  {
    "title": ...,
    "year": ...,
    "annotator_role": ...,
    "time_spent": ...,
    "pairs_by_reviewers": [
      {
        "reviewer": ...,
        "references": ...,
        "reviews": [
          {
            "review": ...,
            "type": "argument" | "claim" | "question",
            "response": ...,
            ...
          }
        ]
      }
    ]
  }
]

Each item in reviews is one review point. Common fields are review for the reviewer text, type for the review-point category, response for the mapped author response, and score_agg for the aggregated human score. Argument-style review points may also include reconstruction with decomposed premises and a conclusion, plus an eval list with premise-level grounding, score, untrivialness, reasoning, and score_agg. For claim and question items, eval is a compact object with annotator scores and optional reasoning, so argument-only fields such as reconstruction, faithfulness, and score_coarse_agg may be absent or null.

Install

pip install -r requirements.txt

Depending on the model backend, the script expects one or more of:

• OPENAI_API_KEY
• ANTHROPIC_API_KEY
• GEMINI_API_KEY
• OPENROUTER_API_KEY

Evaluate Released Outputs

Standard setting:

python run_eval.py --model_name claude-sonnet-4-20250514

Author-response setting:

python run_eval.py \
  --model_name claude-sonnet-4-20250514 \
  --use_author_response

If you want to inspect another released output directory directly, you can still override --save_path.

Run New Inference

Generate new predictions in the standard setting:

python run_eval.py \
  --mode generate \
  --model_name claude-sonnet-4-20250514

Generate new predictions in the author-response setting:

python run_eval.py \
  --mode generate \
  --model_name claude-sonnet-4-20250514 \
  --use_author_response

Outputs are written to:

outputs/outputs_iclr/eval_iclr_<MODEL>.json

or, with --use_author_response:

outputs/outputs_iclr_author_response/eval_iclr_<MODEL>.json

Notes

• The released annotation file already includes the mapped response field for each review point, so the author-response prompts can be used directly.
• The script defaults are now wired to the public iclr dataset and output naming.

Citation

@article{ryu2025reviewscore,
  title={ReviewScore: Misinformed Peer Review Detection with Large Language Models},
  author={Ryu, Hyun and Jang, Doohyuk and Lee, Hyemin S and Jeong, Joonhyun and Kim, Gyeongman and Cho, Donghyeon and Chu, Gyouk and Hwang, Minyeong and Jang, Hyeongwon and Kim, Changhun and others},
  journal={arXiv preprint arXiv:2509.21679},
  year={2025}
}