LM Evaluation Harness
Benchmarks
TaskDatasetSOTA resultTrendResults
LM Evaluation Harness zero-shot and five-shot
72.52HellaSwag Acc
30
LM Evaluation Harness 0-shot
80.66WG
30
LM Evaluation Harness (test)
44.28ARC Challenge Acc
24
LM Evaluation Harness (LM Eval) (test)
74.11WG (Winograd Schema)
22
LM Evaluation Harness MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, CommonsenseQA
70.5MMLU
19
LM Evaluation Harness
72.76MMLU (CoT)
19
lm-evaluation-harness (SciQ, ARC-E, ARC-C, LogiQA, OBQA, BoolQ, HellaSwag, PIQA, WinoGrande) zero-shot
69.4SciQ Accuracy
19
LM Evaluation Harness Downstream Suite (HellaSwag, PIQA, WinoGrande, OpenBookQA, SIQA, BoolQ, TriviaQA, MMLU, ARC-Challenge, ARC-Easy, MathQA, SciQ)
72.52HellaSwag Accuracy
18
LM Evaluation Harness PIQA, HellaSwag, COPA, RTE, OpenBookQA, LAMBADA-OpenAI
75.97Average Score
16
LM Evaluation Harness
60.35Accuracy
16
LM Evaluation Harness MMLU, ARC-Challenge, HellaSwag, TruthfulQA, Winogrande, GSM8K standard
65.8MMLU
16
LM Evaluation Harness NQ, MMLU STEM, ARC, SciQ, LogiQA, BoolQ
29.81NQ Accuracy
15
LM-Evaluation-Harness Commonsense Reasoning: LAMBADA, WikiText, ARC, HellaSwag, PIQA, WinoGrande, BoolQ, SciQ
11.86LAMBADA Perplexity (PPL)
13
LM-Evaluation-Harness ARC, BoolQ, HellaSwag, LAMBADA, PIQA, RACE, SciQ, Record, OBQA
46.8ARC Challenge
13
lm-evaluation-harness suite (HellaSwag, RACE, PIQA, WinoGrande, ARC-e, ARC-c, OBQA)
57.18HellaSwag
12
LM-Evaluation-Harness ARC-c, ARC-e, BoolQ, HellaS., MMLU, OBQA, PIQA, WG
58.4ARC-c Accuracy
12
LM Evaluation Harness
66.6Avg Accuracy
8
LM Evaluation Harness ARC-C, ARC-E, BoolQ, HellaSwag, LAMBADA, PiQA, WinoGrande zero-shot SlimPajama data-restricted scenario
25.76ARC-C
6
LM Evaluation Harness
57.2WG Score
5
lm-evaluation-harness ARC-E, BoolQ, HellaSwag, OBQA, SciQ
0.381ARC-E Accuracy
4
LM Evaluation Harness 0-shot v1.0.0
50.1HellaSwag Accuracy
4