Question Answering on OBQA (test)
92.7AccuracyEnsemble
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| EnsembleParams (M)=11.3052026.06 | 92.7 | 4.96 | 0.27 | |
| MLEParams (M)=3.7682026.06 | 92.25 | 6.4 | 0.38 | |
| MC-DropoutParams (M)=3.7682026.06 | 91.8 | 6.6 | 0.36 | |
| MAPParams (M)=3.7682026.06 | 91.72 | 6.61 | 0.39 | |
| BLoBParams (M)=5.4032026.06 | 91.72 | 2.16 | 0.21 | |
| LaplaceParams (M)=3.7682026.06 | 91.68 | 1.9 | 0.23 | |
| BaRAParams (M)=5.6082026.06 | 91.25 | 2.2 | 0.21 | |
| FVAE-LoRAParams (M)=33.532026.06 | 91.21 | 2.08 | 0.23 | |
| ScalaBLParams (M)=3.7692026.06 | 90.9 | 2.43 | 0.23 | |
| C-LoRAParams (M)=4.2752026.06 | 89.9 | 5.23 | 0.31 | |
| OracleModel Size=Large, Expertise Distribution=Dist. 32026.04 | 71.6 | — | — | |
| OracleModel Size=Large2026.04 | 71.6 | — | — | |
| OracleModel Size=Large, Noise Type=Uniform2026.04 | 71.6 | — | — | |
| REALMModel Size=Large, Expertise Distribution=Dist. 2, Noise Type=Systematic2026.04 | 71 | — | — | |
| REALMModel Size=Large, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 70.68 | — | — | |
| REALMModel Size=Large, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 70.12 | — | — | |
| REALMModel Size=Large, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 70.12 | — | — | |
| REALMModel Size=Large, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 70.12 | — | — | |
| REALMModel Size=Large, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 69.8 | — | — | |
| REALMModel Size=Large, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 69.68 | — | — | |
| noisyModel Size=Large, Expertise Distribution=Dist. 2, Noise Type=Systematic2026.04 | 69.32 | — | — | |
| NOISYModel Size=Large, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 68.8 | — | — | |
| NOISYModel Size=Large, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 65.68 | — | — | |
| NOISYModel Size=Large, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 65.68 | — | — | |
| NOISYModel Size=Large, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 65.4 | — | — | |
| NOISYModel Size=Large, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 64.8 | — | — | |
| NOISYModel Size=Large, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 61.36 | — | — | |
| LLAMA 1Size=65B2023.07 | 60.2 | — | — | |
| LLAMA 2Size=70B2023.07 | 60.2 | — | — | |
| OracleModel Size=Base, Expertise Distribution=Dist. 32026.04 | 59.6 | — | — | |
| OracleModel Size=Base2026.04 | 59.6 | — | — | |
| OracleModel Size=Base, Noise Type=Uniform2026.04 | 59.6 | — | — | |
| REALMModel Size=Base, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 58.68 | — | — | |
| LLAMA 1Size=33B2023.07 | 58.6 | — | — | |
| LLAMA 2Size=7B2023.07 | 58.6 | — | — | |
| REALMModel Size=Base, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 58.6 | — | — | |
| LLAMA 2Size=34B2023.07 | 58.2 | — | — | |
| REALMModel Size=Base, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 57.96 | — | — | |
| REALMModel Size=Base, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 57.96 | — | — | |
| REALMModel Size=Base, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 57.88 | — | — | |
| REALMModel Size=Base, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 57.72 | — | — | |
| LLAMA 1Size=7B2023.07 | 57.2 | — | — | |
| LLAMA 2Size=13B2023.07 | 57 | — | — | |
| NOISYModel Size=Base, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 56.84 | — | — | |
| FalconSize=40B2023.07 | 56.6 | — | — | |
| LLAMA 1Size=13B2023.07 | 56.4 | — | — | |
| NOISYModel Size=Base, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 55.12 | — | — | |
| NOISYModel Size=Base, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 55 | — | — | |
| NOISYModel Size=Base, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 55 | — | — | |
| NOISYModel Size=Base, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 54.36 | — | — | |
| MPTSize=30B2023.07 | 52 | — | — | |
| FalconSize=7B2023.07 | 51.6 | — | — | |
| MPTSize=7B2023.07 | 51.4 | — | — | |
| NOISYModel Size=Base, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 49.12 | — | — | |
| Token DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 47.8 | — | — | |
| Expanded DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 47.8 | — | — | |
| BaselineModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 47.6 | — | — | |
| BaselineModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 47.4 | — | — | |
| Expanded DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 47.2 | — | — | |
| Expert DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.5, Drop Strategy=skips one out of eight experts2025.03 | 46.8 | — | — | |
| Token DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 46.4 | — | — | |
| EntailerBackbone=T5-large, Fine-tuning=none, Fact corpus=provided with dataset2023.05 | 45.8 | — | — | |
| BaselineModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 45.4 | — | — | |
| Expanded DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 45.4 | — | — | |
| Token DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 45.2 | — | — | |
| Expert DropModel=OLMoE-Instruct, Capacity Factor=2.0, Drop Strategy=bottom 10% of lowest-load experts2025.03 | 44.6 | — | — | |
| REALMModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 44.28 | — | — | |
| REALMModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 44.28 | — | — | |
| OracleModel Size=Small, Expertise Distribution=Dist. 32026.04 | 44.2 | — | — | |
| OracleModel Size=Small2026.04 | 44.2 | — | — | |
| OracleModel Size=Small, Noise Type=Uniform2026.04 | 44.2 | — | — | |
| Expanded DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 43.4 | — | — | |
| REALMModel Size=Small, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 42.56 | — | — | |
| REALMModel Size=Small, Expertise Distribution=Dist. 3, Noise Type=Systematic2026.04 | 42.56 | — | — | |
| REALMModel Size=Small, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 42.52 | — | — | |
| REALMModel Size=Small, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 42.52 | — | — | |
| REALMModel Size=Small, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 42.48 | — | — | |
| BaselineModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 42.4 | — | — | |
| Expert DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.0, Drop Strategy=bottom 10% of lowest-load experts2025.03 | 41.8 | — | — | |
| Expert DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.5, Drop Strategy=bottom 10% of lowest-load experts2025.03 | 41.4 | — | — | |
| Token DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 40.4 | — | — | |
| noisyModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 39.52 | — | — | |
| NOISYModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 39.52 | — | — | |
| REALMModel Size=Small, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 39.44 | — | — | |
| NOISYModel Size=Small, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 35.8 | — | — | |
| NOISYModel Size=Small, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 35.8 | — | — | |
| NOISYModel Size=Small, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 35.76 | — | — | |
| NOISYModel Size=Small, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 35 | — | — | |
| noisyModel Size=Small, Expertise Distribution=Dist. 3, Noise Type=Systematic2026.04 | 35 | — | — | |
| NOISYModel Size=Small, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 30.28 | — | — |