Performance Prediction on BIG-bench (test)
1.144Boolean ExpressionsSize
Evaluation Results
| Method | Links | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SizeDescription=Logistic scaling laws using model parameter size (Owen, 2024)2026.02 | 1.144 | 0.687 | 1.022 | 1.764 | 4.411 | 1.277 | 4.394 | 1.326 | 1.328 | 3.926 | 0.783 | 1.112 | 1.682 | 1.816 | 3.26 | 2.041 | 25.289 | 3.31 | 7.084 | 3.561 | |
| OSLDescription=PCA decomposition approach from Ruan et al. (2024)2026.02 | 0.239 | 0.293 | 0.149 | 0.68 | 1.092 | 0.467 | 2.339 | 0.103 | 0.966 | 1.26 | 0.47 | 0.094 | 0.096 | 0.974 | 0.527 | 1.379 | 10.087 | 1.278 | 5.5 | 1.473 | |
| SCDescription=Structured capabilities model with latent factor capability estimates2026.02 | 0.206 | 0.338 | 0.25 | 0.608 | 0.862 | 0.37 | 2.472 | 0.096 | 0.958 | 0.504 | 0.661 | 0.082 | 0.146 | 0.351 | 0.528 | 1.612 | 5.879 | 1.205 | 4.835 | 1.156 |