Calibration on stream 2,000-question
0.08ECEVerbalized + Temp Scaling
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Verbalized + Temp ScalingBackbone=Llama 3.1-8B-Instruct, Temperature Scaling=True (5-fold CV)2026.03 | 0.08 | 0.213 | 0.681 | 64.4 | 0.089 | |
| SECLBackbone=Llama 3.1-8B-Instruct, Adaptation layers=Last 8 layers (rank 8), Tau=3.02026.03 | 0.083 | 0.222 | 0.643 | 64.6 | 0.08 | |
| P(True) Norm + Temp ScalingBackbone=Llama 3.1-8B-Instruct, Temperature Scaling=True (5-fold CV)2026.03 | 0.108 | 0.208 | 0.716 | 64.6 | 0.103 | |
| P(True) NormBackbone=Llama 3.1-8B-Instruct, Temperature Scaling=False2026.03 | 0.12 | 0.211 | 0.718 | 64.6 | 0.117 | |
| VerbalizedBackbone=Llama 3.1-8B-Instruct, Temperature Scaling=False2026.03 | 0.225 | 0.258 | 0.684 | 64.4 | 0.225 | |
| SECLModel=Llama 3.2-3B, Cost=1.8–4.62026.03 | 5 | 24.1 | 58.7 | — | — | |
| SECLModel=Gemma 2-2B, Cost=1.8–4.62026.03 | 5.6 | 25.4 | 54.8 | — | — | |
| P(True) NormModel=Llama 3.2-3B, Cost=62026.03 | 6.5 | 22.3 | 69.4 | — | — | |
| SECLModel=Llama 3.1-8B, Cost=1.8–4.62026.03 | 8.3 | 22.2 | 64.3 | — | — | |
| DINCOModel=Llama 3.2-3B, Cost=∼102026.03 | 10.1 | 20.7 | 76.2 | — | — | |
| DINCOModel=Phi 3.5-Mini, Cost=∼102026.03 | 11 | 21.2 | 74.9 | — | — | |
| SECLModel=Phi 3.5-Mini, Cost=1.8–4.62026.03 | 11 | 25.1 | 52.1 | — | — | |
| DINCOModel=Llama 3.1-8B, Cost=∼102026.03 | 11.7 | 21 | 75.6 | — | — | |
| P(True) NormModel=Llama 3.1-8B, Cost=62026.03 | 12 | 21.1 | 71.8 | — | — | |
| P(True) NormModel=Gemma 2-2B, Cost=62026.03 | 14.1 | 25.9 | 65 | — | — | |
| P(True) NormModel=Phi 3.5-Mini, Cost=62026.03 | 15.4 | 22.7 | 67.5 | — | — | |
| VerbalizedModel=Llama 3.2-3B, Cost=12026.03 | 17 | 29.2 | 51 | — | — | |
| VerbalizedModel=Llama 3.1-8B, Cost=12026.03 | 22.5 | 25.8 | 68.4 | — | — | |
| VerbalizedModel=Phi 3.5-Mini, Cost=12026.03 | 25.1 | 27.5 | 60 | — | — | |
| VerbalizedModel=Gemma 2-2B, Cost=12026.03 | 25.6 | 31.4 | 55.8 | — | — | |
| DINCOModel=Gemma 2-2B, Cost=∼102026.03 | 40.8 | 41 | 56.6 | — | — |