Performance Prediction Suite (GSM8k, MATH, BBH, TriviaQA, MBPP, AGIEval, DROP, MMLU-pro)
1.55Mean Absolute Prediction Error (%)COD (Complete)
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| COD (Complete)subset_to_full_mapping=true2025.02 | 1.55 | 2.68 | 2.68 | 0.79 | 0.47 | 1.97 | 2.42 | 1.64 | 1.05 | 1.39 | |
| COD (w/o mapping)subset_to_full_mapping=false2025.02 | 2.24 | 5.26 | 4.7 | 0.5 | 2.91 | 1.98 | 0.89 | 5.26 | 1.08 | 0.57 | |
| End-to-end(exp)extrapolation_type=exponential-based2025.02 | 3.1 | 6 | 4 | 3.86 | 0.64 | 0.68 | 1.75 | 6 | 4.11 | 3.72 | |
| End-to-end(passrate)trials_per_evaluation_set=1002025.02 | 5.02 | 8.8 | 6.71 | 8.8 | 3.51 | 4 | 7.34 | 6.78 | 0.26 | 2.74 | |
| End-to-end(BNSL)framework=multi-segment power-law2025.02 | 5.17 | 13.05 | 4.23 | 5.88 | 13.05 | 5.86 | 2.55 | 0.82 | 1.53 | 7.42 | |
| Loss-intermediate2025.02 | 5.29 | 9.39 | 9.39 | 6.95 | 2.33 | 5.81 | 5.52 | 1.41 | 5.37 | 5.55 |