Language Model Calibration on APIGen (test)
1.7smECEIsotonic
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Isotonic2026.06 | 1.7 | 84.3 | 69.6 | 79.7 | 78 | |
| HRE2026.06 | 2.3 | 84.1 | 71.7 | 80.5 | 78.8 | |
| Beta2026.06 | 2.4 | 84 | 69.8 | 79.5 | 77.8 | |
| NWKR2026.06 | 2.9 | 84.1 | 70.1 | 79.9 | 78.1 | |
| ACUTE pca20 PlattEstimator=PCA-20, Recalibration=Platt2026.06 | 2.9 | 92.2 | 84.3 | 86.8 | 87.8 | |
| ACUTE pca10 PlattEstimator=PCA-10, Recalibration=Platt2026.06 | 3 | 90.9 | 82.2 | 86.1 | 86.5 | |
| ACUTE pca20 BetaEstimator=PCA-20, Recalibration=Beta2026.06 | 3 | 92.2 | 84.3 | 86.8 | 87.8 | |
| Platt2026.06 | 3.1 | 84.5 | 71.1 | 79.8 | 78.5 | |
| ACUTE pca10 BetaEstimator=PCA-10, Recalibration=Beta2026.06 | 3.2 | 90.8 | 82.2 | 86.1 | 86.4 | |
| ACUTE cosineEstimator=cosine2026.06 | 3.5 | 87.6 | 77.2 | 82.7 | 82.6 | |
| ACUTE cosine PlattEstimator=cosine, Recalibration=Platt2026.06 | 3.5 | 87.4 | 77.3 | 82.8 | 82.5 | |
| ACUTE cosine BetaEstimator=cosine, Recalibration=Beta2026.06 | 3.6 | 87.4 | 77.3 | 82.6 | 82.5 | |
| ACUTE pca10Estimator=PCA-102026.06 | 3.6 | 90.6 | 82.1 | 86.3 | 86.4 | |
| ACUTE pca20 IsotonicEstimator=PCA-20, Recalibration=Isotonic2026.06 | 3.7 | 91.7 | 83.8 | 86.4 | 87.4 | |
| ACUTE pca10 IsotonicEstimator=PCA-10, Recalibration=Isotonic2026.06 | 4.1 | 90.3 | 82.1 | 85.7 | 86 | |
| ACUTE cosine IsotonicEstimator=cosine, Recalibration=Isotonic2026.06 | 4.3 | 87.3 | 76.9 | 82.1 | 82.2 | |
| ACUTE early actLayers=early activations2026.06 | 5 | 90.1 | 81.3 | 85.2 | 85.5 | |
| ACUTE pca20Estimator=PCA-202026.06 | 5.4 | 91.5 | 83.7 | 86.6 | 87.3 | |
| ACUTE mid actLayers=middle activations2026.06 | 6.3 | 90.7 | 83.4 | 87.1 | 87.1 | |
| ACUTE late actLayers=late activations2026.06 | 6.8 | 90.7 | 83.4 | 87 | 87 | |
| Raw Conf2026.06 | 22.2 | 30.5 | 52 | 76.6 | 52.8 |