# On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks

Source: [On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks](<https://www.sota2.com/research/paper/68fa68851e209131cea1caa8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:38:59.898Z

Authors: Stephen Mussmann, Robin Jia, Percy Liang

Published: 2020-10-10T21:56:27.000Z

## Abstract

Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e.g., $99.99\\%$ of examples are negatives). In contrast, many recent datasets heuristically choose examples to ensure label balance. We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only $2.4\\%$ average precision when evaluated on realistically imbalanced test data. We instead collect training data with active learning, using a BERT-based embedding model to efficiently retrieve uncertain points from a very large pool of unlabeled utterance pairs. By creating balanced training data with more informative negative examples, active learning greatly improves average precision to $32.5\\%$ on QQP and $20.1\\%$ on WikiQA.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Is Hired Classification (IH) on English tweets (test)](<https://www.sota2.com/research/sota/is-hired-classification-ih-on-english-tweets-test.md>): is hired classification (IH); English tweets (test); AP: 99.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2010.05103>)

[PDF](<https://arxiv.org/pdf/2010.05103.pdf>)
