You shouldn’t blindly trust benchmarks - you should be able to encode your preferences relative to y...

TL;DR · AI 摘要
ParseBench-2 是一个全面的文档解析基准框架,允许用户根据自身需求定义解析偏好并自定义评估指标。
核心要点
- ParseBench-2 允许用户根据使用场景定义解析偏好,而非依赖单一的基准指标。
- 通过编码人类偏好,ParseBench-2 支持自动化的评估指标设计,提升评估的灵活性和准确性。
- ParseBench-2 提供了对 OCR 工具性能的多维度分析,例如文本线性化、表格保留等。
结构提纲
按章节快速跳转。
- §引言
文章指出不应盲目信任基准,而应根据使用场景定义偏好并设计评估指标。
ParseBench-2 不仅是一个全面的文档解析基准,更是一个用于设计解析评估的框架。
用户可以通过定义偏好,如表格、图表或空行保留,来定制评估指标。
通过编码代理,可以自动化设计评估指标,而非手动制定。
使用 LLM 作为判断工具,可以实现更细致、可重复的评估结果。
ParseBench-2 是一个全面开源的框架,支持持续改进和扩展。
思维导图
用一张图看清主题之间的关系。
查看大纲文本(无障碍 / 无 JS 友好)
- ParseBench-2 框架
- 设计目标
- 支持用户自定义偏好
- 提供多维度解析评估
- 评估方法
- LLM 作为判断工具
- 自动化指标设计
- 应用场景
- 表格与图表解析
- 空行与合并单元格处理
金句 / Highlights
值得收藏与分享的关键句。
You shouldn’t blindly trust benchmarks - you should be able to encode your preferences relative to your use cases and design eval metrics around that.
The important part is not hand-crafting a deterministic parsing metric. Metric design can be automated by coding agents.
One OCR tool might be better at linearizing text, but worse at preserving formatting. Another OCR tool might be better at preserving empty rows in tables but struggle with merged cells.
Jerry Liu on X: "You shouldn’t blindly trust benchmarks - you should be able to encode your preferences relative to your use cases and design eval metrics around that. We’re excited to release ParseBench-2 as not only a comprehensive document parsing benchmark, but also a framework for parsing ben… / X
@jerryjliu0
$
/$
You shouldn’t blindly trust benchmarks - you should be able to encode your preferences relative to your use cases and design eval metrics around that. We’re excited to release ParseBench-2 as not only a comprehensive document parsing benchmark, but also a framework for parsing benchmark design. Where you can articulate a set of human preferences of what “good parsing” looks like and “hillclimb” evaluation metrics against that. The important part is not hand-crafting a deterministic parsing metric. Metric design can be automated by coding agents; what’s important is an open framework + datasets for encoding human preferences and optimizing metrics. 📋 For instance, if you’re dealing with mostly backoffice data, you may care about accurate form parsing, bounding boxes, and scans! 📊 On the other hand, if you’re dealing with investor research reports, you may care more about tables and charts, but less around handwriting. We let you encode a severity policy that encodes the impact of every negative parsing error, and forces the metric to fit against it - think of fitting a linear regression in high-dimensional space. A baseline approach for encoding human preferences is LLM as a judge. But the result of this approach vs. a “simple” LLM as a judge call is that you get much more granular, repeatable, quantifiable breakdowns of each dimension that are efficiently run at scale. For instance, one OCR tool might be better at linearizing text, but worse at preserving formatting. Another OCR tool might be better at preserving empty rows in tables but struggle with merged cells. The scores reflected on our website do reflect a “default” preference, but it is explicitly and transparently articulated both in our Github and technical report. The important part is that you can change and customize it for your needs. ParseBench-2 is our most ambitious, comprehensive open benchmark to date. It is not simply a set of deterministic metrics, but a framework for designing parse evals. We’re rapidly working on improvements and expansions (e.g. fixing minor annotation issues and tuning metrics). If you have thoughts please chime in! Everything is fully OSS. ParseBench-2 Repo:
github.com/run-llama/pars…
ParseBench-2 Blog:
llamaindex.ai/blog
Technical Report:
llamaindex.ai/files/parseben…
00:00
Nov 12
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with
4:24 PM · Nov 13, 2026
·
4,054
Views
30
5
83
20