Multiple Choice Question Answering on MMLU-Pro Engineering (Calibration & Set Prediction)
0.995Calibration Threshold (q-hat)Conformal Social Choice
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Conformal Social ChoiceDebate Round=3, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.995 | 95 | 3.84 | 52.6 | 70 | |
| Conformal Social ChoiceDebate Round=2, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.99 | 94.8 | 3.63 | 50.5 | 84.6 | |
| Conformal Social ChoiceDebate Round=1, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.987 | 95.5 | 3.92 | 45.2 | 96.6 | |
| Conformal Social ChoiceDebate Round=3, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.98 | 90.5 | 2.13 | 67.2 | 92.9 | |
| Conformal Social ChoiceDebate Round=0, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.975 | 93.8 | 4.4 | 26.8 | 96.9 | |
| Conformal Social ChoiceDebate Round=2, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.963 | 88.5 | 2 | 64.3 | 79 | |
| Conformal Social ChoiceDebate Round=1, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.96 | 90.5 | 2.4 | 56.5 | 92.4 | |
| Conformal Social ChoiceDebate Round=0, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.952 | 87.6 | 3.08 | 32 | 96.8 |