# CODAH: An Adversarially Authored Question-Answer Dataset for Common Sense Source: [CODAH: An Adversarially Authored Question-Answer Dataset for Common Sense]() Collection index: [llms.txt]() Data updated: 2025-12-14T17:13:37.347Z Authors: Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fernandez, Doug Downey Published: 2019-04-08T21:25:50.000Z ## Abstract Commonsense reasoning is a critical AI capability, but it is difficult to construct challenging datasets that test common sense. Recent neural question answering systems, based on large pre-trained models of language, have already achieved near-human-level performance on commonsense knowledge benchmarks. These systems do not possess human-level common sense, but are able to exploit limitations of the datasets to achieve human-level scores. We introduce the CODAH dataset, an adversarially-constructed evaluation dataset for testing common sense. CODAH forms a challenging extension to the recently-proposed SWAG dataset, which tests commonsense knowledge using sentence-completion questions that describe situations observed in video. To produce a more difficult dataset, we introduce a novel procedure for question acquisition in which workers author questions designed to target weaknesses of state-of-the-art neural question answering systems. Workers are rewarded for submissions that models fail to answer correctly both before and after fine-tuning (in cross-validation). We create 2.8k questions via this procedure and evaluate the performance of multiple state-of-the-art question answering systems on our dataset. We observe a significant gap between human performance, which is 95.3%, and the performance of the best baseline accuracy of 67.5% by the BERT-Large model. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Lateral Thinking on BRAINTEASER Sentence Puzzle SemEval 2024 Task 9 (test)](): Lateral Thinking; BRAINTEASER Sentence Puzzle SemEval 2024 Task 9 (test); Original Score: 65. - [Lateral Thinking on BRAINTEASER Word Puzzle SemEval 2024 Task 9 (test)](): Lateral Thinking; BRAINTEASER Word Puzzle SemEval 2024 Task 9 (test); Original Score: 43.8. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()