lm-eval-harness
Benchmarks
TaskDatasetSOTA resultTrendResults
LM Eval Harness (HellaSwag, BoolQ, WinoGrande, PiQA, ARC-easy, ARC-challenge) zero-shot
75.46Mean Accuracy
60
LM-Eval-Harness Suite (PIQA, HellaSwag, LAMBADA, ARC-e, ARC-c, SciQ, Race, MMLU) zero-shot
80.7PIQA
32
lm-eval-harness 11 tasks
63.430-shot Accuracy
24
LM-Eval-Harness OBQA, PIQA, ARC-e, ARC-c, HellaSwag, WinoGrande
44.2OBQA Accuracy
14
LM-Eval-Harness
17.8LAMBADA Perplexity (PPL)
10
LM-Eval-Harness MMLU, ARC-Easy, HellaSwag, PIQA, OpenBookQA, WinoGrande
24.3MMLU Accuracy
10
lm-eval-harness PIQA, COPA, OpenBookQA, Winogrande, SciQA, ARC-E, ARC-C
78.8PIQA Accuracy
10
lm-eval-harness (test)
74.22MMLU
9
EleutherAI lm-eval-harness ARC-C, ARC-E, HellaSwag, PIQA, WinoGrande standard (test val)
22.7ARC-Challenge Accuracy
6
LM-Eval-Harness Hungarian
38.6Arc (hu) Acc
4
LM Eval Harness
0.48HellaSwag Accuracy
3