MMLU
Benchmarks
TaskDatasetSOTA resultTrendResults
MMLU
52.7Accuracy (Heterogeneous Knowledge & Reasoning)
3
MMLU
1,317Rollout Time (min)
3
MMLU-Pro
88.93Accuracy
3
MMLU
92.63Accuracy
3
MMLU (test)
48.09Throughput (Tokens/sec)
3
MMLU-Pro Hindi medical
50.7Accuracy
3
MMLU-flex
67.46Accuracy
3
MMLU all-subjects (test)
94.4Accuracy
3
MMLU
0.001RMSE
3
MMLU (test)
32.92Accuracy (Knowledge, test)
3
MMLU Professional Law (test)
80.43Accuracy
3
MMLU HS World History (test)
95.38Accuracy
3
MMLU College Medicine (test)
0.864Accuracy
3
MMLU hi
24.8MMLU hi Accuracy
3
MMLU en
23.5Accuracy
3
MMLU
24.31Accuracy
3
MMLU
80.38Mean Accuracy (±STD)
3
MMLU
78.32Mean Accuracy
3
MMLU
48.7MMLU Overall Accuracy
3
MMLU
86Accuracy
3
MMLU Mathematics and History Error + Cost setting
31.2Deferral Ratio (7B, Str)
3
MMLU Mathematics and History Error Only setting
99.67B (Str) Deferral Ratio
3
MMLU cross-benchmark
77.4Accuracy
3
MMLU c
29.87Normalized Accuracy
3
MMLU-Pro zero-shot latest
69.65Accuracy (zero-shot)
3