# Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization

Source: [Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization](<https://www.sota2.com/research/paper/6a17cc37dbd7150506f8146e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-29T05:00:12.873Z

Authors: Beiduo Chen, Pingjun Hong, Ziyun Zhang, Benjamin Roth, Anna Korhonen, Barbara Plank

Published: 2026-05-27T17:55:00.000Z

## Abstract

Free-text explanations extend human label variation (HLV) beyond label disagreement by revealing the reasoning and preferences behind annotators' decisions. We study whether large language models (LLMs) can learn and reproduce such annotator-specific label-explanation behavior. Using two sentence-pair tasks with four annotators each -- natural language inference and paraphrase judgment -- we first analyze whether annotators exhibit stable individual patterns. We find that such patterns are weak at the single-annotation level due to strong input-content effects, but become detectable after input-content reduction and annotator-level aggregation. We then compare prompting and supervised fine-tuning (SFT) baselines and propose cross-annotator preference optimization (CAPO), which contrasts a target annotator's response with other valid but less target-specific annotations for the same input. Experiments show that prompting is limited and unstable, SFT better captures annotator-specific behavior, and CAPO further improves aggregation-aware imitation and judge-based attribution while preserving target-specific reasoning patterns under human validation. Overall, our results show that HLV can be learned as annotator-specific label-explanation behavior, suggesting a path toward scalable explanation-based annotation grounded in annotator histories rather than labels alone.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Simulation Performance on R2](<https://www.sota2.com/research/sota/simulation-performance-on-r2.md>): Simulation Performance; R2; Accuracy: 46.

- [Variation error identification and evaluation on VariErr (Expanded)](<https://www.sota2.com/research/sota/variation-error-identification-and-evaluation-on-varierr-expanded.md>): Variation error identification and evaluation; VariErr (Expanded); Accuracy: 63.7.

- [Human validation of generated explanations on VariErr 50-sample](<https://www.sota2.com/research/sota/human-validation-of-generated-explanations-on-varierr-50-sample.md>): Human validation of generated explanations; VariErr 50-sample; Make-Sense Rate: 46.

- [Annotator Simulation on VariErr](<https://www.sota2.com/research/sota/annotator-simulation-on-varierr.md>): Annotator Simulation; VariErr; Label Accuracy: 62.7.

- [Explanation Generation on VariErr](<https://www.sota2.com/research/sota/explanation-generation-on-varierr.md>): Explanation Generation; VariErr; SFT (Mean Feature KL): not recorded.

- [Explanation Generation on R2](<https://www.sota2.com/research/sota/explanation-generation-on-r2.md>): Explanation Generation; R2; SFT (Mean Feature KL): not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.28802>)

[PDF](<https://arxiv.org/pdf/2605.28802.pdf>)
