Evaluation Suite
Benchmarks
TaskDatasetSOTA resultTrendResults
5-level evaluation suite 1.0 (full)
83.8PPA (Macro)
7
Evaluation Suite Zero-shot (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA, MTQA) v1 (test)
82.14BoolQ Accuracy (Zero-shot)
6
Zero-shot Evaluation Suite (ARC, HellaSwag, MMLU, PIQA, WinoGrande)
47.7ARC Challenge Accuracy
6
Zero-shot Evaluation Suite (ARC, HellaSwag, MMLU, PIQA, Winogrande)
48.1ARC-C Acc
6
7-task evaluation suite (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-E, ARC-C, OBQA) (test)
62.9BoolQ Accuracy
4
Evaluation Suite Zero-shot (ARCC, ARCE, BOOLQ, HSWAG, PIQA)
70.83Average Accuracy
4
Table 3 Evaluation Suite
70.94Average Score
4
Evaluation Suite Combined (held-out)
48.4Accuracy
4