FinCriticalED

A Visual Benchmark for Financial Fact-Level OCR Evaluation

Yueru He, Xueqing Peng^*, Yupeng Cao, Yan Wang, Lingfei Qian, Haohang Li, Yi Han, Shuyao Wang, Ruoyu Xiang, Fan Zhang, Zhuohan Xie, Mingquan Lin, Prayag Tiwari, Jimin Huang, Guojun Xiong, Sophia Ananiadou

^*Corresponding author, xueqing.peng2024@gmail.com

Paper · Data · Website

---

Overview

FinCriticalED (Financial Critical Error Detection) is a fact-centric visual benchmark for evaluating OCR and vision-language systems on whether they faithfully preserve decision-critical evidence in financial documents. It contains 859 real-world financial document pages paired with ground-truth HTML, with 9,481 expert-annotated facts spanning five financially critical field types: Numbers, Monetary Units, Temporal Data, Reporting Entities, and Financial Concepts. The accompanying evaluation suite combines critical-field-aware metrics with a context-aware protocol, and is used to benchmark 13 OCR pipelines, OCR-native models, open-source VLMs, and proprietary MLLMs.

Abstract

Recent progress in multimodal large language models (MLLMs) has substantially improved document understanding, yet strong OCR performance on surface metrics does not necessarily imply faithful preservation of decision-critical evidence. This limitation is especially consequential in financial documents, where small visual errors, such as a missing negative marker, shifted decimal point, incorrect unit scale, or misaligned reporting date, can induce materially different interpretations.

To study this gap, we introduce FinCriticalED (Financial Critical Error Detection), a fact-centric visual benchmark for evaluating OCR and vision-language systems through the lens of evidence fidelity in high-stakes document understanding. FinCriticalED contains 859 real-world financial document pages paired with ground-truth HTML, with 9,481 expert-annotated facts spanning five financially critical field types: Numbers, Monetary Units, Temporal Data, Reporting Entities, and Financial Concepts.

We further develop an evaluation suite, including critical-field-aware metrics and a context-aware protocol, to assess whether model outputs preserve financially critical facts beyond lexical similarity. We benchmark 13 OCR pipelines, OCR-native models, open-source VLMs, and proprietary MLLMs on FinCriticalED. Results show that conventional OCR metrics can substantially overestimate factual reliability, and that OCR-specialized systems may outperform much larger general-purpose MLLMs in preserving critical financial evidence under complex layouts. FinCriticalED provides a rigorous benchmark for trustworthy financial OCR and a broader testbed for high-stakes multimodal document understanding.

Results

Model performance on the FinCriticalED benchmark:

Model	Size	General (%)			Fact-Level (%)
		R1	RL	E↓	N-FFA	T-FFA	M-FFA	R-FFA	FC-FFA	FFA
OCR Pipelines
MinerU2.5	1.2B	95.71	95.30	6.02	98.76	96.48	54.05	91.09	96.44	94.64
PP-OCRv5	0.07B	97.54	96.55	3.10	95.7	90.29	90.00	86.62	93.75	91.91
Specialized OCR VLMs
DeepSeekOCR	3B	94.73	94.42	7.33	93.47	91.96	83.53	92.27	94.36	92.67
DeepSeekOCR-2	6B	92.90	92.18	10.72	82.63	91.9	82.83	88.69	86.51	86.19
GLM-OCR	0.9B	95.10	94.74	6.43	93.24	98.53	88.89	97.84	100.00	96.92
Open-source MLLMs
Gemma-3n-E4B-it	4B	83.49	79.59	23.82	52.65	77.06	64.71	74.65	72.86	65.68
Qwen3-VL-8B-Instruct	8B	97.68	97.40	2.93	98.47	96.99	97.65	93.18	99.24	96.88
Llama-4-Maverick	17B	98.00	97.62	3.70	97.77	97.99	97.65	94.26	98.48	96.48
Qwen3.5-397B-A17B	397B	98.12	98.00	2.59	87.72	87.99	86.14	91.22	94.4	89.70
Proprietary MLLMs
GPT-4o	-	90.40	88.35	16.01	59.56	84.59	81.92	81.78	70.84	71.68
GPT-5	-	91.81	89.56	15.79	66.83	94.48	92.35	89.19	91.77	81.65
Claude-Sonnet-4.6	-	98.84	98.73	1.69	98.59	97.99	97.06	94.02	98.94	97.23
Gemini-2.5-Pro	-	98.81	98.37	2.46	97.24	97.82	97.65	94.18	98.94	96.74

R1 = ROUGE-1, RL = ROUGE-L, E↓ = Edit Distance (lower is better), FFA = Fact-level Financial Accuracy. Best results per column in bold.

Usage

1. Running models

1. Configure the MODELS list in model_eval/main.py by uncommenting the model(s) you want to run.

2. Create a .env file in model_eval/ with the relevant API keys:

   OPENAI_API_KEY=...
   ANTHROPIC_API_KEY=...
   GOOGLE_API_KEY=...
   TOGETHER_API_KEY=...
   ZAI_API_KEY=...

3. Run main.py to generate model OCR output:

   cd model_eval
   python main.py

• Results are saved to results/{model_tag}_zero-shot/pred_{i}.txt. Already-completed samples are skipped on re-runs.
• To limit the number of samples for a quick test, set max_samples in main(), e.g. max_samples = 10.

Supported models

Cloud VLMs (API key required, no local setup):

Model key in MODELS	Provider	API key
gpt-4o	OpenAI	OPENAI_API_KEY
gpt-5	OpenAI	OPENAI_API_KEY
claude-sonnet-4-6	Anthropic	ANTHROPIC_API_KEY
gemini-2.5-pro	Google	GOOGLE_API_KEY
meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8	Together AI	TOGETHER_API_KEY
Qwen/Qwen2.5-VL-72B-Instruct	Together AI	TOGETHER_API_KEY
glm-ocr	ZAI	ZAI_API_KEY

Local OCR pipelines (no API key, local setup required):

Model key in MODELS	Description
paddleocrv5	PP-OCRv5 plain text OCR; outputs raw text lines
monkeyocr	MonkeyOCR; requires a running local server
mineru	MinerU2.5; runs locally via Hugging Face
deepseekocr	DeepSeek-OCR; runs locally via Hugging Face
deepseekocr2	DeepSeek-OCR-2; runs locally via Hugging Face

Setting up paddleocr and paddleocrv5-ppstructure

Linux or WSL is required; PaddlePaddle has limited Windows support.

Install dependencies:

pip install paddleocr paddlepaddle
# For paddleocrv5-ppstructure, also install:
pip install shapely pyclipper scikit-image imutils lmdb

Model weights (~200 MB) are downloaded automatically on first run.

• paddleocrv5 runs PP-OCRv5 locally on CPU and outputs plain text. Some samples may raise a PaddlePaddle oneDNN/PIR compatibility error; these are caught and skipped automatically. No configuration is needed beyond the install above.
• paddleocrv5-table uses TableRecognitionPipelineV2 to detect tables and output them as HTML (<table>/<tr>/<td>), with all other text as <p> tags, producing a full HTML document per page. Note: PPStructureV3 requires CUDA and is not supported in CPU-only environments.

Setting up monkeyocr

MonkeyOCR requires a running HTTP server. Start the server separately before running main.py (see the MonkeyOCR repository for server setup instructions).

Set the server URL via an environment variable (defaults to http://localhost:8000):

MONKEYOCR_API_URL=http://your-server:8000

2. Running evaluation

Traditional OCR metrics

After running main.py, run evaluation.py to compute ROUGE-1, ROUGE-L, and Edit Distance:

python evaluation.py

Results are saved as results/{model_tag}_zero-shot_rouge1_eval.csv.

LLM-as-Judge

In llm-as-a-judge-prompt.py, GPT-4o serves as the evaluator: it extracts financial entities (Numbers, Dates, Monetary Units, etc.) from the ground-truth HTML and verifies their presence in the model-generated HTML. The LLM judge performs normalization, contextual matching, and fine-grained fact checking under a structured evaluation prompt.

Resources on Hugging Face

Resource	Description
FinCriticalED collection	FinCriticalED resources on Hugging Face
TheFinAI/FinCriticalED	Benchmark dataset: 859 financial document pages with ground-truth HTML and expert-annotated facts

Citation

If you find this work useful, please cite:

@misc{he2025fincriticaledvisualbenchmarkfinancial,
      title={FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR Evaluation}, 
      author={Yueru He and Xueqing Peng and Yupeng Cao and Yan Wang and Lingfei Qian and Haohang Li and Yi Han and Ruoyu Xiang and Mingquan Lin and Prayag Tiwari and Jimin Huang and Guojun Xiong and Sophia Ananiadou},
      year={2025},
      eprint={2511.14998},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.14998}, 
}

License

The code in this repository is released under the MIT License. Datasets and models on Hugging Face keep their own licenses, stated on each card.

---

Built by The Fin AI · Hugging Face · GitHub