Multiple Choice Question Answering on PubMedQA (test)
76.03AccuracyKAPPA
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| KAPPAModel=Llama 3.1 8B, Intervention Layers=62025.09 | 76.03 | 96.97 | 0.599 | |
| BaseModel=Llama 3.1 8B2025.09 | 75.73 | 96.43 | 0.434 | |
| KAPPAModel=Llama 3.1 8B, Intervention Layers=32025.09 | 75.33 | 98.57 | 0.566 | |
| KAPPAModel=Llama 3.1 8B, Intervention Layers=12025.09 | 75.17 | 96.77 | 0.576 | |
| CAAModel=Llama 3.1 8B2025.09 | 75.1 | 94.37 | 0.467 | |
| DoLAModel=Llama 3.1 8B2025.09 | 74.63 | 92.53 | 0.536 | |
| KAPPAModel=Qwen 2.5 7B, Intervention Layers=62025.09 | 73 | 90.7 | 0.588 | |
| KAPPAModel=Qwen 2.5 7B, Intervention Layers=12025.09 | 72.57 | 91.43 | 0.565 | |
| KAPPAModel=Qwen 2.5 7B, Intervention Layers=32025.09 | 72.53 | 91.37 | 0.565 | |
| BaseModel=Qwen 2.5 7B2025.09 | 72.33 | 89.8 | 0.57 | |
| CAAModel=Qwen 2.5 7B2025.09 | 72.27 | 89.63 | 0.57 | |
| Knowledge Probe (KP)Model=Qwen 2.5 7B2025.09 | 72 | 100 | 0 | |
| DoLAModel=Qwen 2.5 7B2025.09 | 71.63 | 89.63 | 0.57 | |
| OracleModel Size=Large, Noise Type=None, Expertise Distribution=Dist. ①2026.04 | 71.25 | — | — | |
| REALMModel Size=Large, Noise Type=Systematic, Expertise Distribution=Dist. ①2026.04 | 62.1 | — | — | |
| NOISYModel Size=Large, Noise Type=Systematic, Expertise Distribution=Dist. ①2026.04 | 61.05 | — | — | |
| OracleModel Size=Base, Noise Type=None, Expertise Distribution=Dist. ①2026.04 | 61 | — | — | |
| REALMModel Size=Large, Noise Type=Asymmetric, Expertise Distribution=Dist. ①2026.04 | 58.9 | — | — | |
| OracleModel Size=Small, Noise Type=None, Expertise Distribution=Dist. ①2026.04 | 58 | — | — | |
| REALMModel Size=Large, Noise Type=Uniform, Expertise Distribution=Dist. ①2026.04 | 57.65 | — | — | |
| REALMModel Size=Base, Noise Type=Systematic, Expertise Distribution=Dist. ①2026.04 | 55.3 | — | — | |
| NOISYModel Size=Small, Noise Type=Systematic, Expertise Distribution=Dist. ①2026.04 | 54.2 | — | — | |
| REALMModel Size=Small, Noise Type=Systematic, Expertise Distribution=Dist. ①2026.04 | 53.7 | — | — | |
| NOISYModel Size=Base, Noise Type=Systematic, Expertise Distribution=Dist. ①2026.04 | 53.1 | — | — | |
| REALMModel Size=Base, Noise Type=Uniform, Expertise Distribution=Dist. ①2026.04 | 51.8 | — | — | |
| REALMModel Size=Base, Noise Type=Asymmetric, Expertise Distribution=Dist. ①2026.04 | 50.7 | — | — | |
| REALMModel Size=Small, Noise Type=Uniform, Expertise Distribution=Dist. ①2026.04 | 50.6 | — | — | |
| REALMModel Size=Small, Noise Type=Asymmetric, Expertise Distribution=Dist. ①2026.04 | 48.8 | — | — | |
| NOISYModel Size=Large, Noise Type=Uniform, Expertise Distribution=Dist. ①2026.04 | 43.85 | — | — | |
| NOISYModel Size=Base, Noise Type=Uniform, Expertise Distribution=Dist. ①2026.04 | 40.6 | — | — | |
| NOISYModel Size=Base, Noise Type=Asymmetric, Expertise Distribution=Dist. ①2026.04 | 36.35 | — | — | |
| NOISYModel Size=Small, Noise Type=Uniform, Expertise Distribution=Dist. ①2026.04 | 35.95 | — | — | |
| NOISYModel Size=Large, Noise Type=Asymmetric, Expertise Distribution=Dist. ①2026.04 | 35.95 | — | — | |
| NOISYModel Size=Small, Noise Type=Asymmetric, Expertise Distribution=Dist. ①2026.04 | 28.8 | — | — |