Multiple Choice Question Answering on MMLU-Pro Chemistry (Calibration & Set Prediction)
0.99Calibration Threshold (q-hat)Conformal Social Choice
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Conformal Social ChoiceDebate Round=3, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.99 | 95.6 | 1.81 | 78.8 | 90 | |
| Conformal Social ChoiceDebate Round=2, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.987 | 95.9 | 1.75 | 77 | 97.1 | |
| Conformal Social ChoiceDebate Round=1, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.982 | 95.8 | 1.91 | 70.8 | 95.5 | |
| Conformal Social ChoiceDebate Round=0, Alpha=0.05, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.962 | 95 | 2.57 | 43.3 | 97.5 | |
| Conformal Social ChoiceDebate Round=0, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.784 | 89 | 1.37 | 65.7 | 93.8 | |
| Conformal Social ChoiceDebate Round=3, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.733 | 87.8 | 1.01 | 98.8 | 42.9 | |
| Conformal Social ChoiceDebate Round=1, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.702 | 88 | 1.06 | 92.8 | 78.4 | |
| Conformal Social ChoiceDebate Round=2, Alpha=0.10, Model Ensemble=Haiku + DeepSeek-R1 + Qwen-3 32B2026.04 | 0.669 | 88 | 1.02 | 97.5 | 77.8 |