PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
arXiv · HuggingFace · ▲39
Abstract (verbatim)
We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.
Background
Background Analysis
Technical Context: Multimodal Large Language Models (MLLMs) are increasingly used in applications like image understanding and content generation (e.g., AI assistants analyzing screenshots or verifying reports), where the core demand is "human-level perceptual reliability"—the ability to accurately interpret complex visual information (e.g., chart data or spatial relationships). For instance, medical imaging requires precise lesion localization, but current benchmarks fail to validate such capabilities effectively.
Previous Limitations: Two major flaws plague existing evaluation systems: (1) Insufficient data coverage, as most benchmarks use simple or narrowly scoped images (e.g., single-object classification), causing models to rely on linguistic priors rather than true visual understanding; (2) Distorted scoring mechanisms, where traditional metrics (e.g., CLIPScore) mask local errors via linear averaging. For example, a model misinterpreting a table number might still score high, even though such errors are "zero-tolerant" in real-world scenarios. This creates a paradox where high leaderboard scores do not translate to reliable real-world performance.
Proposed Solution: The paper introduces PerceptionRubrics, addressing these issues through three steps: (1) Curating a dataset of 1,038 high-information images and generating "golden captions" via a "circular peer-review" process (iterated by state-of-the-art MLLMs and verified by humans); (2) Extracting 12,000+ atomic rubrics categorized into "Must-Right" (critical facts) and "Easy-Wrong" (common hallucinations/omissions); (3) Implementing a "gated scoring" mechanism where violating "Must-Right" rules triggers binary penalties, aligning scores with human sensitivity to severe errors.
Key Difference: Unlike prior work, PerceptionRubrics shifts focus from "holistic semantic matching" to "fine-grained perceptual auditing." By extracting rules from model errors and using non-linear scoring (distinguishing fatal vs. minor errors), it bridges the gap between benchmark performance and real-world reliability. This approach not only reveals gaps in complex domains (e.g., an 8% perceptual deficit between open-source and proprietary models) but also provides interpretable diagnostics for future improvements.
