Reasoning Suite
Benchmarks
TaskDatasetSOTA resultTrendResults
Reasoning Suite Zero-shot (PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c) (val test)
76.55Average Accuracy
305
Reasoning Suite (ARC-e, ARC-c, HellaSwag, PIQA, Winogrande) zero-shot
6,540Average Reasoning Score
125
Reasoning Suite Zero-shot Aggregate
73.2Aggregate Score
50
Reasoning Suite Average
74.8Accuracy
45
Reasoning Suite (ARC, HellaSwag, PIQA, WinoGrande, MMLU, OpenBookQA, Real-world QA) Zero-shot Llama-3.1-8B-Instruct with Alpaca calibration
9.63PPL
32
Reasoning Suite Zero-shot (BoolQ, WinoG., PIQA, OBQA, HellaS., ARC-e, ARC-c)
82.63BoolQ Accuracy
24
Reasoning Suite Zero-shot (PIQA, Hella Swag, LAMBADA, ARC-e, ARC-c, SciQ, Race, MMLU)
80.7PIQA
21
Reasoning Suite PiQA, LAMBDA, ARC, HellaSwag
62.69PiQA Score
20
Reasoning Suite Zero-shot (ArcC, ArcE, PiQA, Wino)
50.43Arc Challenge (C) Accuracy
16
Reasoning Suite Zero-shot (ARC-e, ARC-c, WG, BQ, PIQA, HS, OBQA, HQA)
49.7ARC-e Accuracy
9
Reasoning Suite Zero-shot (PIQA, ARC, HS, WG, BoolQ, MMLU)
80.2PIQA Accuracy
9
Reasoning Suite Zero-shot (ARC-E, BoolQ, HSwag, LAMBADA, OBQA, PIQA, SocIQA, WinoGr.)
45.5ARC-E Accuracy
9
Reasoning Suite
85.12GSM8K
9
Reasoning Suite (OBQA, PIQA, ARC-e, ARC-c, Hella, Wino) Zero-shot
46.8Accuracy on OBQA (Zero-shot)
8
Reasoning Suite BBH, GPQA, MuSR
83.4BBH
7
Reasoning Suite (MMLU-Pro, GPQA Diamond, AIME-24, AIME-25) zero-shot
74.8MMLU-Pro Accuracy
6
Reasoning Suite Zero-shot (PIQA, ARCe, ARCc, BoolQ, Hella., Wino.)
79.05PIQA Accuracy
4
Reasoning Suite
89.36BIG-Bench Hard
4