Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering
About
We propose a simple refactoring of multi-choice question answering (MCQA) tasks as a series of binary classifications. The MCQA task is generally performed by scoring each (question, answer) pair normalized over all the pairs, and then selecting the answer from the pair that yield the highest score. For n answer choices, this is equivalent to an n-class classification setup where only one class (true answer) is correct. We instead show that classifying (question, true answer) as positive instances and (question, false answer) as negative instances is significantly more effective across various models and datasets. We show the efficacy of our proposed approach in different tasks -- abductive reasoning, commonsense question answering, science question answering, and sentence completion. Our DeBERTa binary classification model reaches the top or close to the top performance on public leaderboards for these tasks. The source code of the proposed approach is available at https://github.com/declare-lab/TEAM.
Key insights
How does binary classification reformulate multi-choice QA?
Instead of scoring all answer choices jointly via softmax, TEAM classifies each (question, answer) pair independently as positive or negative, training a two-node classification head on a transformer encoder.
What distinguishes TEAM from prior score-based methods?
TEAM replaces n-class softmax scoring with per-pair binary classification. Across ten benchmarks with RoBERTa and DeBERTa, it improves on most datasets but underperforms on SIQA, PIQA, and CICERO v1 where choices are lexically very similar.
Where could this approach be practically useful?
TEAM enables on-the-fly candidate evaluation without requiring all answers upfront, potentially bridging multi-choice QA and open-ended generation. The authors note this flexibility but document only benchmark evaluations, not production deployments.
What limitations do the authors acknowledge?
TEAM still assumes candidate answers exist, lacking full open-ended flexibility. It underperforms score-based methods on datasets with highly similar choices. The authors leave identifying a principled method-selection criterion for future work.