A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

About

This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. In addition to being one of the largest corpora available for the task of NLI, at 433k examples, this corpus improves upon available resources in its coverage: it offers data from ten distinct genres of written and spoken English--making it possible to evaluate systems on nearly the full complexity of the language--and it offers an explicit setting for the evaluation of cross-genre domain adaptation.

Adina Williams, Nikita Nangia, Samuel R. Bowman · 2017

Related benchmarks

Task	Dataset	Result
Natural Language Inference	SNLI (test)	Accuracy81.5	694
Sentiment Classification	Stanford Sentiment Treebank SST-2 (test)	Accuracy87.2	105
Natural Language Inference	MultiNLI matched (test)	Accuracy67.5	65
Natural Language Inference	MultiNLI Mismatched	Accuracy69.4	60
Natural Language Inference	MultiNLI mismatched (test)	Accuracy67.6	56
Factual Consistency Evaluation	SummaC	CGS46	52
Natural Language Inference	MultiNLI Matched	Accuracy69.8	49
Factual Consistency Evaluation	QAGS XSUM	Spearman Correlation0.7	39
Factual Consistency Evaluation	QAGS CNNDM	Spearman Correlation-16.4	38
Factual Consistency Evaluation	TRUE benchmark	PAWS (AUC-ROC)83.11	37