List your company
ResearchFormerly WizWand

Boolean Question Answering on BoolQ (Accuracy)

88.91AccuracyQwen 3
35.786849.578463.3777.1616Aug 20, 2025Oct 11, 2025Dec 2, 2025Jan 24, 2026Mar 17, 2026May 8, 2026Jun 30, 2026
Updated 4mo ago

Evaluation results

MethodLinks
Qwen 3Best alpha=0.47, Strategy=Best Pairwise2026.0588.91
BaselineBackbone=Qwen3-MoE-30B-A3B, Type=–, Storage=61.06GB, Evaluation Protocol=zero-shot2026.0688.7
Llama 3.1Best alpha=0.41, Strategy=Best Pairwise2026.0588.43
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M25%, Storage=46.56GB, Evaluation Protocol=zero-shot2026.0688.2
GRAPEModel=Mixtral-8x22B, Scope=Global, Number of experts pruned (e)=2e2026.0488
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.0688
OriginalModel=Mixtral-8x22B, Scope=None2026.0487.9
DEKModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.0487.6
EnumerateModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.0487.4
Gemma 3Best alpha=0.35, Strategy=Best Pairwise2026.0585.74
Hard-Routed MoR-LoRABackbone=Llama-8B, # Trainable Parameters=≈ 109M2026.0684.43
OursQBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.0684.22
OursQBackbone=Qwen3-MoE-30B-A3B, Type=P25% Q4b, Storage=11.64GB, Evaluation Protocol=zero-shot2026.0684.22
GRAPEModel=Mixtral-8x22B, Scope=Global, Number of experts pruned (e)=4e2026.0484.1
LoRAMixer TopK=1Backbone=Llama-8B, # Trainable Parameters=≈ 1.133B2026.0683.7
OursBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.0683.36
DEKModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.0483.1
Hard-Routed MoR-LoRABackbone=Llama-3B, # Trainable Parameters=≈ 73M2026.0683.09
LoRAMixer TopK=2 NormalizedBackbone=Llama-3B, # Trainable Parameters=≈ 606M2026.0682.87
LoRAMixer TopK=2 NormalizedBackbone=Llama-8B, # Trainable Parameters=≈ 1.133B2026.0682.4
Count-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.0481.5
EnumerateModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.0480.5
BaselineBackbone=Qwen1.5-MoE-A2.7B, Type=–, Storage=28.67GB, Evaluation Protocol=zero-shot2026.0679.57
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.0679.54
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=7.10GB, Evaluation Protocol=zero-shot2026.0679.36
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.0679.2
OursBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.0678.93
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.0678.6
GRAPEModel=DeepSeek-MoE, Scope=Global, Number of experts pruned (e)=2e2026.0477.8
GRAPEModel=DeepSeek-MoE, Scope=Global, Number of experts pruned (e)=4e2026.0477.5
DEKModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=2e2026.0477.4
OriginalModel=DeepSeek-MoE, Scope=None2026.0477.2
C-PruneBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.0677.12
LoRAMixer TopK=1Backbone=Llama-3B, # Trainable Parameters=≈ 606M2026.0676.67
DEKModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=4e2026.0476.6
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.0676.57
Count-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=2e2026.0476.2
Count-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=4e2026.0475.9
EAC-MoEBackbone=Qwen1.5-MoE-A2.7B, Type=P38% Q3.03b, Storage=4.35GB, Evaluation Protocol=zero-shot2026.0675.35
MoE-I2Backbone=Qwen1.5-MoE-A2.7B, Type=L53.98%, Storage=15.18GB, Evaluation Protocol=zero-shot2026.0675.08
MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.0674.86
Count-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.0474.4
Router-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=4e2026.0472.1
Router-guidedModel=DeepSeek-MoE, Scope=Local, Number of experts pruned (e)=2e2026.0471.3
MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.0669.37
MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.0669.05
He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.0668.41
He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.0663.55
Router-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=2e2026.0462.8
Router-guidedModel=Mixtral-8x22B, Scope=Local, Number of experts pruned (e)=4e2026.0462.7
CoFReevaluation protocol=likelihood-based answer scoring2026.0560.21
SSAArchitecture=Nemotron-style decoder, Parameters=114M, Training steps=22K, Training dataset=FineWeb, Zero-shot=true2025.0856.18
MDLMevaluation protocol=likelihood-based answer scoring2026.0549.42
WandaBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.0640.55
WandaBackbone=Qwen3-MoE-30B-A3B, Type=P25%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.0638.5
SoftmaxArchitecture=Nemotron-style decoder, Parameters=114M, Training steps=22K, Training dataset=FineWeb, Zero-shot=true2025.0837.83