Accuracy on Big-Bench Hard (BBH)
82.3AccuracyMPO+
Evaluation Results
| Method | Links | |
|---|---|---|
| MPO+Backbone=olmo-3.1-32b-it-sft2026.04 | 82.3 | |
| Pair+Backbone=olmo-3.1-32b-it-sft2026.04 | 82.1 | |
| DPOBackbone=olmo-3.1-32b-it-sft2026.04 | 81.6 | |
| Mag+Backbone=olmo-3.1-32b-it-sft2026.04 | 81.6 | |
| Soft+Backbone=olmo-3.1-32b-it-sft2026.04 | 81.6 | |
| RFTBackbone=olmo-3.1-32b-it-sft2026.04 | 77.6 | |
| Pair+Backbone=olmo-3-7b-it-sft2026.04 | 73.8 | |
| MPO+Backbone=olmo-3-7b-it-sft2026.04 | 73.6 | |
| Mag+Backbone=olmo-3-7b-it-sft2026.04 | 73.2 | |
| Soft+Backbone=olmo-3-7b-it-sft2026.04 | 72.9 | |
| DPOBackbone=olmo-3-7b-it-sft2026.04 | 71.5 | |
| RFTBackbone=olmo-3-7b-it-sft2026.04 | 66.7 | |
| LoRA# Trainable Parameters=3407.87K, Backbone=Llama-3.1-8B2025.07 | 60.39 | |
| (IA)3# Trainable Parameters=524.29K, Backbone=Llama-3.1-8B2025.07 | 60.29 | |
| BaseBackbone=olmo-3.1-32b-it-sft2026.04 | 60 | |
| SITE (M = 50)# Trainable Parameters=1.02K, Backbone=Llama-3.1-8B, M=502025.07 | 58.04 | |
| SITE (M = 1)# Trainable Parameters=1.02K, Backbone=Llama-3.1-8B, M=12025.07 | 56.76 | |
| MPO+Backbone=gemma-3-4b-sft2026.04 | 51.7 | |
| Soft+Backbone=gemma-3-4b-sft2026.04 | 51.4 | |
| I2CL# Trainable Parameters=0.13K, Backbone=Llama-3.1-8B2025.07 | 50.6 | |
| DPOBackbone=gemma-3-4b-sft2026.04 | 50.5 | |
| Pair+Backbone=gemma-3-4b-sft2026.04 | 50 | |
| RFTBackbone=gemma-3-4b-sft2026.04 | 49.9 | |
| Mag+Backbone=gemma-3-4b-sft2026.04 | 49.3 | |
| GRPOBackbone=Qwen2.5-Math-7B2025.10 | 47.7 | |
| BaseBackbone=Qwen2.5-Math-7B2025.10 | 47.5 | |
| SePTBackbone=Qwen2.5-Math-7B2025.10 | 47.3 | |
| 10-shot ICLBackbone=Llama-3.1-8B2025.07 | 47.17 | |
| BaseBackbone=olmo-3-7b-it-sft2026.04 | 46 | |
| DenseBase Model=Llama-3.2-3B, Model architecture type=Dense, Few-shot CoT=true2025.06 | 44.1 | |
| MTV# Trainable Parameters=1.02K, Backbone=Llama-3.1-8B2025.07 | 42.54 | |
| MoBBase Model=Llama-3.2-3B, Model architecture type=MoB, Few-shot CoT=true2025.06 | 42.2 | |
| TVBackbone=Llama-3.1-8B2025.07 | 42.01 | |
| MiCRoBase Model=Llama-3.2-3B, Model architecture type=MiCRo, Few-shot CoT=true2025.06 | 42 | |
| BaseBackbone=gemma-3-4b-sft2026.04 | 40.4 | |
| Prompt Tuning# Trainable Parameters=81.92K, Backbone=Llama-3.1-8B2025.07 | 33.45 | |
| Original ModelBackbone=Llama-3-8B, Avg. Sparsity (L/H)=0% / 0%, Avg. Mem (GB)=4.50, Time (s)=34522026.06 | 30.5 | |
| DenseBase Model=Llama-3.2-1B, Model architecture type=Dense, Few-shot CoT=true2025.06 | 30.4 | |
| MiCRoBase Model=Llama-3.2-1B, Model architecture type=MiCRo, Few-shot CoT=true2025.06 | 29.8 | |
| Learning to AllocateBackbone=Llama-3-8B, Avg. Sparsity (L/H)=34% / 24%, Avg. Mem (GB)=4.10, Time (s)=23802026.06 | 29.8 | |
| FlexiDepthBackbone=Llama-3-8B, Avg. Sparsity (L/H)=32% / –, Avg. Mem (GB)=4.05, Time (s)=24202026.06 | 28.2 | |
| MoBBase Model=SmollM2-360M, Model architecture type=MoB, Few-shot CoT=true2025.06 | 27.7 | |
| Teacher (Qwen3-0.6B)Role=Teacher, Model=Qwen3-0.6B, Evaluation Protocol=generation-based evaluation2026.03 | 27.4 | |
| MoBBase Model=Llama-3.2-1B, Model architecture type=MoB, Few-shot CoT=true2025.06 | 27.4 | |
| DenseBase Model=SmollM2-360M, Model architecture type=Dense, Few-shot CoT=true2025.06 | 27.3 | |
| MiCRoBase Model=SmollM2-360M, Model architecture type=MiCRo, Few-shot CoT=true2025.06 | 27.2 | |
| AdaSkipBackbone=Llama-3-8B, Avg. Sparsity (L/H)=35% / –, Avg. Mem (GB)=3.88, Time (s)=23502026.06 | 26.8 | |
| IVBackbone=Llama-3.1-8B2025.07 | 26.25 | |
| MiCRoBase Model=SmollM2-135M, Model architecture type=MiCRo, Few-shot CoT=true2025.06 | 25.4 | |
| Oracle Static PruningBackbone=Llama-3-8B, Avg. Sparsity (L/H)=33% / 20%, Avg. Mem (GB)=3.80, Time (s)=22502026.06 | 24.5 | |
| DenseBase Model=SmollM2-135M, Model architecture type=Dense, Few-shot CoT=true2025.06 | 24.1 | |
| MoBBase Model=SmollM2-135M, Model architecture type=MoB, Few-shot CoT=true2025.06 | 23.5 | |
| Static PruningBackbone=Llama-3-8B, Avg. Sparsity (L/H)=33% / 0%, Avg. Mem (GB)=3.92, Time (s)=23102026.06 | 21.3 | |
| Hybrid KDASequence Mixer=KDA, Attention Layers=7, Evaluation Protocol=generation-based evaluation2026.03 | 18.9 | |
| FVBackbone=Llama-3.1-8B2025.07 | 17.82 | |
| 0-shot ICLBackbone=Llama-3.1-8B2025.07 | 16.41 | |
| LIVE# Trainable Parameters=131.10K, Backbone=Llama-3.1-8B2025.07 | 12.89 | |
| Hybrid MambaSequence Mixer=Mamba2, Attention Layers=7, Evaluation Protocol=generation-based evaluation2026.03 | 12.1 | |
| Pure MambaSequence Mixer=Mamba2, Attention Layers=0, Evaluation Protocol=generation-based evaluation2026.03 | 8.4 | |
| Pure KDASequence Mixer=KDA, Attention Layers=0, Evaluation Protocol=generation-based evaluation2026.03 | 5.4 |