# It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance

Source: [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](<https://www.sota2.com/research/paper/68fa6f701e209131cea4590e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:20:27.449Z

Authors: Arjun Subramonian, Xingdi Yuan, Hal Daum\\'e III, Su Lin Blodgett

Published: 2023-05-15T21:12:07.000Z

## Abstract

Progress in NLP is increasingly measured through benchmarks; hence, contextualizing progress requires understanding when and why practitioners may disagree about the validity of benchmarks. We develop a taxonomy of disagreement, drawing on tools from measurement modeling, and distinguish between two types of disagreement: 1) how tasks are conceptualized and 2) how measurements of model performance are operationalized. To provide evidence for our taxonomy, we conduct a meta-analysis of relevant literature to understand how NLP tasks are conceptualized, as well as a survey of practitioners about their impressions of different factors that affect benchmark validity. Our meta-analysis and survey across eight tasks, ranging from coreference resolution to question answering, uncover that tasks are generally not clearly and consistently conceptualized and benchmarks suffer from operationalization disagreements. These findings support our proposed taxonomy of disagreement. Finally, based on our taxonomy, we present a framework for constructing benchmarks and documenting their limitations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.09022>)

[PDF](<https://arxiv.org/pdf/2305.09022.pdf>)
