English | 中文

---

Why DocParsingBench?

DocParsingBench is an evaluation toolkit purpose-built for intelligent document parsing products. We open-source the evaluation dataset, complete evaluation methodology, and analysis tools so developers can compare document parsing models in a unified, reproducible, and diagnosable way.

• Data from real production scenarios: All evaluation data is collected from real business scenarios, covering five industries: finance, legal, scientific research, manufacturing, and education, with 14 types of complex production documents.
• Open dataset and evaluation methodology: DocParsingBench provides public data, a unified evaluation pipeline, and reproducible scoring, allowing different models to be compared with the same yardstick.
• More complete visual analysis: Built-in visualization tools help locate exactly where a model fails, and support score breakdowns by business scenario to identify which document types a model handles reliably.
• Faster evaluation: On the same model, OmniDocBench takes about 720s, OLMbench takes about 480s, while DocParsingBench takes only about 125s, making it better suited for frequent iteration and batch comparison.
• Higher model compatibility: DocParsingBench evaluates the final Markdown output end to end without relying on any specific intermediate format, so most document parsing models can be plugged into the benchmark.
• More flexible scoring logic: Segment matching and the Hungarian algorithm make the evaluation tolerant of multiple reasonable reading-order expressions, reducing false penalties caused by assuming a single canonical order.

If this project helps you, please consider giving it a ⭐ Star in the top-right corner. Your support is a huge encouragement to the team.

Latest Updates

[2026.04.17] DocParsingBench evaluation toolkit release. It will provide unified scoring for the three core elements of document parsing: text, formula, and table, along with CLI batch evaluation, segment matching, visualization analysis, and leaderboard generation. 📊

[2026.03.09] DocParsingBench dataset release. The first intelligent document parsing dataset built for real industry scenarios, covering finance, legal, scientific research, manufacturing, and education. Now available on Hugging Face、ModelScope！🔥🔥🔥

Dataset

We systematically collected and annotated document samples from real business workflows, preserving real-world noise such as scan artifacts, stamp occlusion, and blurry characters. The dataset contains 1400 pages, covering Chinese, English, and mixed Chinese-English documents; layouts include single-column, double-column, triple-column, and mixed layouts. Annotations use Markdown, and chemical structures follow the SoMarkdown specification, combining SMILES with LaTeX to ensure complete rendering.

Industry	Typical Documents	Key Characteristics
Finance	Brokerage research reports, annual reports of listed companies, prospectuses	Multi-column tables, stamped scanned documents, complex table structures
Legal	Legal documents, contract clauses, industry standards	Standardized headers and footers, dense footnotes
Scientific Research	Academic papers, programming textbooks, patent full texts	Double-column/multi-column mixed layouts, many formulas, code blocks, chemical equations
Manufacturing	Operation SOPs, forms, invoices and receipts	Blurry scans, handwritten fields, QR/barcode interference
Education	English, chemistry, and mathematics textbooks	Chemical structures, reaction equations, multiple-choice questions, fill-in-the-blank questions
Overall Coverage	1400 pages of real business documents	Chinese, English, and mixed Chinese-English; single-column, double-column, triple-column, and mixed layouts

Evaluation Performance

DocParsingBench significantly reduces overall evaluation time while preserving multi-dimensional evaluation for text, formulas, and tables; FastCDM further improves formula scoring throughput through a lighter formula rendering pipeline. Below are the evaluation performance results of olmOCR-2-7B on the three benchmarks.

1. Evaluation Time Across Three Benchmarks

Benchmark	Evaluation Items	Count	Total Time (s)	Time per Item (s)	Speed Ratio
OmniDocBench	Text + Formula CDM + Table TEDS	1,638 files	977	0.6	1x
olmOCR-bench	Math + Absent + Present + Order + Table	1,403 files	703	0.5	1.19x
DocParsingBench	Text NED + Formula CDM + Table TEDS	1,342 files	106.2	0.079	7.54x

2. CDM vs FastCDM Speed Comparison

Method	Rendering Engine	Formula Count	Total Time (s)	Formulas/s	Time per Item (s)	Speedup
CDM	TeX Live + ImageMagick	1,000	608	1.64	0.61	1x
FastCDM	Chrome + KaTeX	1,000	97	10.29	0.097	6.26x

Visual Analysis Tool

DocParsingBench provides a per-sample visual analysis interface that places the original page, ground-truth Markdown, predicted Markdown, and evaluation metrics in one view. Developers can filter results by industry, sub-industry, and sample to quickly locate errors across text, formulas, tables, and other document elements.

Metric Overview

Given two Markdown files (prediction and ground truth), DocParsingBench performs segment-level matching and scoring by category, and outputs both overall and per-category scores with reusable metric wrappers and visualization tools.

• Segment categories: text (with inline formulas), display_formula, table, image (currently dropped in evaluation)
• Segmentation: text and display formulas are split by line boundaries; tables are bounded by <table> ... </table>
• Matching: Hungarian matching is applied within each category using configured matching metrics (NED, CDM, TEDS)
• Metric wrappers: NED/CER, CDM, TEDS/TEDS-S
• Overall metric: DPB (Document Parsing Benchmark), a weighted average with default weights α=0.5, β=0.3, γ=0.2

$$\begin{aligned} text\_score &= \alpha \cdot avg(1 - NED) + (1 - \alpha) \cdot avg(CDM) \\\ display\_formula\_score &= avg(CDM) \\\ table\_score &= avg(TEDS) \\\ DPB &= \alpha \cdot text\_score + \beta \cdot display\_formula\_score + \gamma \cdot table\_score \end{aligned}$$

Evaluation Leaderboard

Rank	Methods	DPB	Text	Formula	Table
1	PaddleOCR-1.5	0.8535	0.8959	0.7527	0.7104
2	MonkeyOCR-Pro-3B	0.8260	0.8669	0.7206	0.7014
3	MinerU2.5	0.8164	0.8426	0.7993	0.7557
4	Qwen3-VL-235B-Instruct	0.7971	0.8496	0.4355	0.6691
5	ChandraOCR-2	0.7906	0.8361	0.7772	0.7242
6	Deepseek-OCR-2	0.7403	0.7917	0.6775	0.5741
7	GLM-OCR	0.7348	0.7695	0.5773	0.5046
8	dots.ocr-1.5	0.6564	0.6885	0.6236	0.5655
9	HunyuanOCR	0.5128	0.5319	0.6018	0.6428

Summary Chart

Interactive Leaderboard

Installation

• Requires Python 3.8+
• Dependencies are declared in pyproject.toml

git clone https://github.com/SoMarkAI/DocParsingBench.git
cd docparsingbench

pip install .

# For local development:
pip install -e .

Configuration

Configuration is defined in YAML and maps 1:1 to the internal Config dataclass. A reference file is provided at config.example.yaml.

Key options:

• chromedriver_path: if unset or null, fastcdm uses its own default.
  • See the FastCDM chromedriver installation guide
• visualize: whether to generate CDM visualization images during evaluation (effective only when formula.metric: "CDM"). Output images are saved in <output>/cdm_vis/.

Local Development With fastcdm Source

To use local fastcdm source code instead of an installed package, set FASTCDM_SRC to the source root:

export FASTCDM_SRC=/path/to/fastcdm

You can add this line to ~/.zshrc or ~/.bashrc for persistence. If not set, the installed fastcdm package is used.

Usage

Evaluation

dpb is packaged as a CLI entrypoint and is equivalent to python -m docparsingbench.cli.

python -m dpb eval \
  --gt path/to/gt.md \
  --pred path/to/pred.md \
  --config config.yaml \
  --out result.json

If --gt and --pred are directories, matching filenames are evaluated in batch.

# gt_dir contains a.md, b.md, c.md ...
# pred_dir contains a.md, b.md, c.md ...
python -m dpb eval \
  --gt gt_dir/ \
  --pred pred_dir/ \
  --config config.yaml \
  --out result.json

After eval, the terminal prints a model-level one-line summary with Model, Files, DPB, Text, Formula, Table, FormulaRenderFailures, and Output.

Segment Testing

python -m dpb segment \
  --in path/to/md \
  --out segments.json

Visualization

dpb visualize \
  --labels path/to/labels.json \
  --img path/to/images_dir \
  --gt path/to/gt_markdowns_dir \
  --pred path/to/pred_markdowns_dir \
  --result path/to/model.result.json

• labels.json stores only sample-to-industry/sub-industry mappings. If --labels is omitted, it is auto-generated alongside gt.

Summary Bar Chart (summary-chart)

dpb summary-chart \
  --labels path/to/labels.json \
  --results path/to/results_dir \
  --exclude-model-prefix deepseek_ocr \
  --y-min 30 \
  --y-max 100 \
  --output path/to/summary_chart.png

• Optional: repeat --exclude-model-prefix to hide model families by result filename prefix.
• Optional: set y-axis range via --y-min / --y-max (defaults: 30 / 100).

Batch evaluation can auto-generate the chart when all conditions are met:

• --gt and --pred are both directories
• summary_chart.enable: true (default: true)
• summary_chart.y_min / summary_chart.y_max (defaults: 30 / 100)
• --labels is omitted and can be auto-generated from gt

dpb eval \
  --gt data/gt/DocParsingBench/markdowns \
  --pred data/pred/some_model_md \
  --config config.yaml \
  --out data/results/some_model_md.result.json

Interactive HTML Leaderboard (leaderboard-html)

Generates a single self-contained .html file with interactive sorting and filtering. Open it in any browser or share it directly without a server.

dpb leaderboard-html \
  --labels path/to/labels.json \
  --results path/to/results_dir \
  --output leaderboard.html \
  --exclude-model-prefix deepseek_ocr   # optional, repeatable

• All data (All + per-industry views) is embedded inline as JSON
• Industry switch: All / Education / Finance / Legal / Manufacturing / Research
• Metrics and ranking in one table: DPB / Text / Formula / Table
• Default sort: DPB descending; click any column header to cycle desc/asc
• Hover a metric cell → cursor-following tooltip with the 4-decimal raw value
• Save as image button exports the current view as PNG via html2canvas
• Smooth bar-width transitions when switching industries or sort columns

Metric Notes

• NED (Normalized Edit Distance): normalized edit distance computed after character-level normalization
• CER (Character Error Rate): edit distance divided by GT length
• CDM: formula matching metric based on fastcdm, returns F1/recall/precision (F1 is used by default)
• TEDS/TEDS-S: table tree-edit-distance-based similarity (greater is better); TEDS-S compares structure only
• Hungarian matching: one-to-one matching within each segment category; unmatched pairs are scored as 0 similarity

DPB Calculation

• Text: text_score = α * avg(1 - NED) + (1 - α) * avg(CDM)
• Display formula: avg(CDM) (or NED depending on config)
• Table: avg(TEDS) (or TEDS-S)

DPB = α * text_score + β * display_formula_score + γ * table_score

Different domain presets (for example paper/finance/tech) can define different weight presets.

Model Runner Scripts

The scripts/ directory provides OCR model runner scaffolding with a unified pipeline: scan image directory -> call model -> post-process -> output Markdown.

Usage

# Deepseek-OCR example
python -m scripts.deepseek_ocr ./images ./output/deepseek_ocr_md

Add a New Model

Inherit BaseModelRunner and implement parse_md:

from scripts.base import BaseModelRunner

class MyModelRunner(BaseModelRunner):
    name = "my_model"

    def parse_md(self, img_path: str) -> str:
        # call model API / SDK / local inference and return markdown
        ...

    def postprocess(self, md: str) -> str:
        # optional: cleanup / formatting
        return md

The base class handles image scanning, tqdm progress display, resume behavior (skip existing outputs), and failure statistics.

Implemented Models

Script	Model	Status
deepseek_ocr.py	DeepSeek OCR	Implemented
dots_ocr.py	Dots OCR	Implemented
glm_ocr.py	GLM OCR	Implemented
hunyuan_ocr.py	Hunyuan OCR	Implemented
mineru.py	MinerU	Implemented
monkey_ocr.py	Monkey OCR Pro 3B	Implemented
paddle.py	PaddleOCR	Implemented
qwen3_vl.py	Qwen3-VL	Implemented
chandra_ocr.py	Chandra OCR	Implemented

Performance Evaluation Design

This project reserves hooks and a unified output schema for performance benchmarking. Real model invocation can be driven externally.

• In CLI eval, when perf.enable=true, it records:
  • segmentation time, matching time, each metric's time, and total time
  • document count and throughput (docs/s)
• Output is written to perf in result.json:
  • phases: timing by phase
  • throughput: document throughput
  • notes: external model invocation marker (empty by default or filled by upper layers)

Benchmark speed reporting should use evaluation-phase runtime + document throughput, excluding external model generation latency. External model latency should be recorded by upper-layer systems.