Boolean Question Answering on BoolQ (Accuracy)
88.91AccuracyQwen 3
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen 3Best alpha=0.47, Strategy=Best Pairwise2026.05 | 88.91 | |
| BaselineBackbone=Qwen3-MoE-30B-A3B, Type=–, Storage=61.06GB, Evaluation Protocol=zero-shot2026.06 | 88.7 | |
| Llama 3.1Best alpha=0.41, Strategy=Best Pairwise2026.05 | 88.43 | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M25%, Storage=46.56GB, Evaluation Protocol=zero-shot2026.06 | 88.2 | |
| GRAPEModel=Mixtral-8x22B, Scope=Global, Number of experts pruned (e)=2e2026.04 | 88 | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.06 | 88 | |
| OriginalModel=Mixtral-8x22B, Scope=None2026.04 | 87.9 | |
| DEKModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.04 | 87.6 | |
| EnumerateModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.04 | 87.4 | |
| Gemma 3Best alpha=0.35, Strategy=Best Pairwise2026.05 | 85.74 | |
| Hard-Routed MoR-LoRABackbone=Llama-8B, # Trainable Parameters=≈ 109M2026.06 | 84.43 | |
| OursQBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 84.22 | |
| OursQBackbone=Qwen3-MoE-30B-A3B, Type=P25% Q4b, Storage=11.64GB, Evaluation Protocol=zero-shot2026.06 | 84.22 | |
| GRAPEModel=Mixtral-8x22B, Scope=Global, Number of experts pruned (e)=4e2026.04 | 84.1 | |
| LoRAMixer TopK=1Backbone=Llama-8B, # Trainable Parameters=≈ 1.133B2026.06 | 83.7 | |
| OursBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.06 | 83.36 | |
| DEKModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.04 | 83.1 | |
| Hard-Routed MoR-LoRABackbone=Llama-3B, # Trainable Parameters=≈ 73M2026.06 | 83.09 | |
| LoRAMixer TopK=2 NormalizedBackbone=Llama-3B, # Trainable Parameters=≈ 606M2026.06 | 82.87 | |
| LoRAMixer TopK=2 NormalizedBackbone=Llama-8B, # Trainable Parameters=≈ 1.133B2026.06 | 82.4 | |
| Count-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.04 | 81.5 | |
| EnumerateModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.04 | 80.5 | |
| BaselineBackbone=Qwen1.5-MoE-A2.7B, Type=–, Storage=28.67GB, Evaluation Protocol=zero-shot2026.06 | 79.57 | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.06 | 79.54 | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=7.10GB, Evaluation Protocol=zero-shot2026.06 | 79.36 | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.06 | 79.2 | |
| OursBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 78.93 | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 78.6 | |
| GRAPEModel=DeepSeek-MoE, Scope=Global, Number of experts pruned (e)=2e2026.04 | 77.8 | |
| GRAPEModel=DeepSeek-MoE, Scope=Global, Number of experts pruned (e)=4e2026.04 | 77.5 | |
| DEKModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=2e2026.04 | 77.4 | |
| OriginalModel=DeepSeek-MoE, Scope=None2026.04 | 77.2 | |
| C-PruneBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 77.12 | |
| LoRAMixer TopK=1Backbone=Llama-3B, # Trainable Parameters=≈ 606M2026.06 | 76.67 | |
| DEKModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=4e2026.04 | 76.6 | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.06 | 76.57 | |
| Count-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=2e2026.04 | 76.2 | |
| Count-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=4e2026.04 | 75.9 | |
| EAC-MoEBackbone=Qwen1.5-MoE-A2.7B, Type=P38% Q3.03b, Storage=4.35GB, Evaluation Protocol=zero-shot2026.06 | 75.35 | |
| MoE-I2Backbone=Qwen1.5-MoE-A2.7B, Type=L53.98%, Storage=15.18GB, Evaluation Protocol=zero-shot2026.06 | 75.08 | |
| MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.06 | 74.86 | |
| Count-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.04 | 74.4 | |
| Router-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=4e2026.04 | 72.1 | |
| Router-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=2e2026.04 | 71.3 | |
| MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 69.37 | |
| MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 69.05 | |
| He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.06 | 68.41 | |
| He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.06 | 63.55 | |
| Router-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.04 | 62.8 | |
| Router-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.04 | 62.7 | |
| CoFReevaluation protocol=likelihood-based answer scoring2026.05 | 60.21 | |
| SSAArchitecture=Nemotron-style decoder, Parameters=114M, Training steps=22K, Training dataset=FineWeb, Zero-shot=true2025.08 | 56.18 | |
| MDLMevaluation protocol=likelihood-based answer scoring2026.05 | 49.42 | |
| WandaBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.06 | 40.55 | |
| WandaBackbone=Qwen3-MoE-30B-A3B, Type=P25%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.06 | 38.5 | |
| SoftmaxArchitecture=Nemotron-style decoder, Parameters=114M, Training steps=22K, Training dataset=FineWeb, Zero-shot=true2025.08 | 37.83 |