SciLitBench

SciLitBench is a benchmark and artifact release for evaluating large language models across three stages of systematic literature review automation:

1. title and abstract screening (TA);
2. full-text screening (FT); and
3. data extraction (DE).

The release contains benchmark labels, prompts, schemas, model predictions, scoring code, audit materials, and raw and harmonized DE annotations. It supports reproduction of the result tables reported in SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews.

Reproduce the reported metrics

The prediction files are distributed as scilitbench-predictions.tar.gz on the repository's Releases page rather than stored in Git history. Download that asset and extract it in the repository root so that the resulting directory is preds/. Its SHA-256 checksum is recorded in release-assets.sha256.

Create a Python environment and install the core dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run the main TA, FT, closed-field DE, and open-field DE result tables:

python code/reproduce.py

Run the main tables plus the LLM-as-a-judge (LAAJ) and calibration diagnostics:

python code/reproduce.py --all

The reproduction commands use the released predictions and annotations. They do not require access to a model server or the source PDFs.

Individual result scripts can also be run directly:

python code/numbers_ta.py
python code/numbers_ft.py
python code/numbers_de_closed_fields.py
python code/numbers_de_laaj_uncalibrated.py
python code/numbers_de_calibration.py
python code/numbers_de_eval_res_limit.py
python code/numbers_de_open_fields.py

Reproducibility scope

The supported workflow recomputes the reported metrics from the released benchmark labels, model predictions, LAAJ outputs, and calibration labels.

The repository also preserves the original TA, FT, and DE generation scripts:

• code/run_ta.py
• code/run_ft.py
• code/run_de_extract_from_starting_schema.py
• code/run_de_schema_iterations.py
• code/run_de_extract_from_final_schema.py
• code/run_de_laaj.py

These scripts record the generation pipeline but are not supported public entry points. They require separately obtained full-text PDFs, local model-serving infrastructure, additional generation dependencies, and site-specific configuration. The release therefore supports reproduction of the reported metrics from released outputs, not regeneration of every model output from the repository alone.

Benchmark data

The main benchmark files are:

File	Contents
benchmark/ta_gold.csv	Human-labeled title and abstract screening set
benchmark/ta_silver.csv	Silver title and abstract screening set
benchmark/ft_gold.csv	Human-labeled full-text screening set
benchmark/ft_silver.csv	Silver full-text screening set
benchmark/de_raw.json	Raw human DE annotations for 888 papers
benchmark/de_harmonized.json	Harmonized human DE annotations for the same papers

The DE fields are year, domain, review_stage, approach, evaluation_results, and limitations.

Human harmonization dimensions and machine harmonization artifacts are under harmonization/. See harmonization/HARMONIZATION_NOTES.md for the released structure and validation notes.

Model outputs

After the prediction asset is extracted, the preds/ directory contains:

• TA predictions under preds/ta/;
• FT few-shot, system-prompt, and two-step predictions under preds/ft/;
• final-schema DE reports under preds/de/reports_final_schema/;
• frontier-model closed-field DE reports under preds/de/frontier_reports_final_schema/;
• parsed open-field DE outputs under preds/de/laaj_json_extractor_outputs/;
• LAAJ outputs under preds/de/laaj_judge_outputs/;
• human LAAJ calibration labels in preds/de/calibration_labels.jsonl; and
• starting-schema reports and schema-refinement artifacts under preds/de/reports_starting_schema/ and preds/de/schema_refinements/.

Data audits

The data_audit/ directory contains the released TA and FT audit samples and labels. Reproduce the reported TA and FT intra-annotator agreement with:

python data_audit/ta_audit_agreement.py
python data_audit/ft_audit_agreement.py

The interactive helper can be used to create a new set of TA audit labels without overwriting the released labels:

python data_audit/ta_audit_labeler.py \
  --input data_audit/ta_audit_sample.csv \
  --output data_audit/ta_audit_labels_new.csv

The TA and FT annotation manual is available at docs/TA_FT_annotation_manual.md.

Full-text availability

The repository does not contain article PDFs. The file full_text_availability.csv records the conservative availability and license metadata review for all 888 full-text papers. It identifies 409 papers for which an open or licensed version was found, 169 with conditional or restrictive terms, and 310 without a validated open or licensed version. These records are metadata only; no article PDF is redistributed, regardless of status. See docs/FULL_TEXT_AVAILABILITY.md for field definitions and release status.

Repository layout

Path	Contents
benchmark/	TA, FT, and DE benchmark annotations
code/	Metric reproduction and historical generation code
code/data/	TA and FT helper tables used by the result scripts
data_audit/	Audit samples, labels, and audit scripts
docs/	Annotation and release documentation
harmonization/	Human and machine harmonization artifacts
preds/	Released TA, FT, and DE model outputs
schemas/	Final DE extraction schemas

Licensing

SciLitBench code and original SciLitBench data are covered by separate licenses. Article titles, abstracts, PDFs, and other third-party material remain subject to their original terms. See LICENSES.md, LICENSE-CODE, LICENSE-DATA, and THIRD_PARTY_NOTICES.md before redistributing material from this repository.

Citation

The paper citation and archival identifier will be added when the preprint record is public.