List your company
ResearchFormerly WizWand

Commonsense Reasoning on MMLU

78.95AccuracyIAP-mv
6.51425.319544.12562.9305Sep 30, 2024Dec 31, 2024Apr 2, 2025Jul 4, 2025Oct 4, 2025Jan 4, 2026Apr 7, 2026
Updated 3mo ago

Evaluation results

MethodLinks
IAP-mvBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0978.95
VanillaBackbone=Qwen3-MoE-30B-A3B, Sparsity=0%2025.1177.8
IAP-ssBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0977.07
Prompt #5Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0976.84
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.1176.6
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.1175.1
Prompt #4Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0974.21
WandaBackbone=Qwen3-MoE-30B-A3B, Sparsity=2:42025.1172.1
Llama-3.1-8BArchitecture=Transformer, Tokens (B)=15000, Evaluation Protocol=5-shot2025.0568
VanillaBackbone=Mixtral-8x7B-v0.1, Sparsity=0%2025.1167.9
PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.1166.8
D2Backbone=Qwen3-MoE-30B-A3B, Sparsity=20%2025.1166.2
PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.1165.7
Falcon3-Mamba-7BArchitecture=Subquadratic (Pretrained), Tokens (B)=7300, Evaluation Protocol=5-shot2025.0565
Sub-MoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.1165
AMV (#1-7)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0963.86
STUNBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.1163.3
WandaBackbone=Mixtral-8x7B-v0.1, Sparsity=2:42025.1162
Prompt #2Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0961.93
VanillaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=0%2025.1161
HC-SMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.1160.8
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%2025.1160.4
Llamba-8BArchitecture=Subquadratic (Distilled), Tokens (B)=12, Evaluation Protocol=5-shot2025.0560
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2025.1160
IAP-mvBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0959.65
HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.1159.4
Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.1159
NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.1158.1
LoLA-8B (ours)Architecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=5-shot, η=64, λ=642025.0557.6
Prompt #7Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0957.19
Prompt #6Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0956.67
IAP-ssBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0956.49
Sub-MoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.1156
WandaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=2:42025.1155.8
Prompt #1Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0955.79
Prompt #6Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0955.25
RecurrentGemma-9BArchitecture=Subquadratic (Pretrained), Tokens (B)=2000, Evaluation Protocol=5-shot2025.0555.1
LoLCATs-8BArchitecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=5-shot2025.0554.9
IAP-mvBackbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0954.39
Self-discoverBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0952.63
Prompt #5Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0952.63
AMV (all)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0952.53
AMV (all)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0952.46
D2Backbone=Qwen1.5-MoE-A2.7B, Sparsity=20%2025.1152.4
Prompt #4Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0952.11
HC-SMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%2025.1151
Prompt #2Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0950.35
Prompt #2Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0950.18
HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.1149
Prompt #4Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0948.95
Mamba2-8BArchitecture=Subquadratic (Pretrained), Tokens (B)=3500, Evaluation Protocol=5-shot2025.0548.7
Prompt #3Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0948.25
Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.1148
D2Backbone=Qwen3-MoE-30B-A3B, Sparsity=40%2025.1147.9
NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.1147.3
Prompt #5Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0946.67
Prompt #3Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0944.56
Prompt #3Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0943.51
Prompt #9Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0943.33
Mamba2-Llama3-8B (L3.1-Instr.)Architecture=Subquadratic (Distilled), Tokens (B)=20, Evaluation Protocol=5-shot2025.0543.2
Prompt #1Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0942.48
Prompt #7Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0942.28
AMV (#1-7)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0941.23
HC-SMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.1139.6
Griffin 7BArchitecture=Subquadratic (Pretrained), Tokens (B)=300, Evaluation Protocol=5-shot2025.0539.3
VanillaBackbone=Deepseek-MoE-16b, Sparsity=0%2025.1137.8
D2Backbone=Qwen1.5-MoE-A2.7B, Sparsity=40%2025.1137.5
In-Place TTTArchitecture=Full Attention, Parameter Count=4B2026.0437.42
PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.1137.2
Prompt #1Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0937.02
PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.1136.9
In-Place TTTArchitecture=Sliding-Window Attention (SWA), Parameter Count=4B2026.0436.48
Full AttentionModel Group=Baselines, Parameter Count=4B2026.0436.43
Sliding-Window Attention (SWA)Model Group=Baselines, Parameter Count=4B2026.0436.06
Prompt #6Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0935.79
Hawk 7BArchitecture=Subquadratic (Pretrained), Tokens (B)=300, Evaluation Protocol=5-shot2025.0535
WandaBackbone=Deepseek-MoE-16b, Sparsity=2:42025.1131
Prompt #9Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0930.7
Mamba-8BArchitecture=Subquadratic (Pretrained), Tokens (B)=1100, Evaluation Protocol=5-shot2025.0528
Sub-MoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.1127
Prompt #8Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0926.32
HC-SMoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.1124.5
Hedgehog-8B (Llama-3)Architecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=5-shot2025.0524.2
HC-SMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2025.1123.2
HC-SMoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.1123
Prompt #8Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0922.56
Sub-MoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.1122
Prompt #9Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0920.18
Prompt #7Backbone=LLaMA-3 70B, Setting=Zero-shot CoT2024.0914.56
Prompt #8Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.099.3