# RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics

Source: [RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics](<https://www.sota2.com/research/paper/69cf49680636e5bf4afcd8bb>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-23T05:00:05.534Z

Authors: Zhengyang Qi, Charles Dickens, Derek Pham, Amanda Dsouza, Armin Parchami, Frederic Sala, Paroma Varma

Published: 2026-04-01T20:34:43.000Z

## Abstract

Rubric-based evaluation is widely used in LLM benchmarks and training pipelines for open-ended, less verifiable tasks. While prior work has demonstrated the effectiveness of rubrics using downstream signals such as reinforcement learning outcomes, there remains no principled way to diagnose how a rubric itself fails from such aggregated or downstream signals alone. To address this gap, we introduce RIFT: RubrIc Failure mode Taxonomy, a taxonomy for systematically characterizing failure modes in rubric composition and design. RIFT consists of eight failure modes organized into three high-level categories: Reliability Failures, Content Validity Failures, and Consequential Validity Failures. RIFT is developed using grounded theory by iteratively annotating rubrics drawn from five diverse data sources spanning general instruction following, code generation, creative writing, and expert-level deep research, until no new failure modes are identified. We evaluate the consistency of the taxonomy by measuring agreement among independent human annotators, observing fair agreement overall (87% pairwise agreement and 0.64 average Cohen's kappa). Finally, to support scalable diagnosis, we propose automated rubric quality metrics and show that they align with human failure-mode annotations, achieving up to 0.925 F1.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Automated failure mode detection alignment on RIFT (test)](<https://www.sota2.com/research/sota/automated-failure-mode-detection-alignment-on-rift-test.md>): Automated failure mode detection alignment; RIFT (test); Subjectivity Score: 86.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.01375>)

[PDF](<https://arxiv.org/pdf/2604.01375.pdf>)