Common Sense Reasoning on HellaSwag (Accuracy)
91.84AccuracyMistral-Small
Evaluation Results
| Method | Links | |
|---|---|---|
| Mistral-SmallModel Size=24B, GPU Hour=∼1.6M2025.05 | 91.84 | |
| Qwen2.5-InstructModel Size=14B, GPU Hour=∼1.8M2025.05 | 88.28 | |
| MiniLogitModel Size=14B, GPU Hour=2202025.05 | 88.25 | |
| InfiGFusionModel Size=14B, GPU Hour=1952025.05 | 88.22 | |
| InfiFusionModel Size=14B, GPU Hour=1602025.05 | 87.91 | |
| FuseLLMModel Size=14B, GPU Hour=2252025.05 | 87.81 | |
| Pivot-SFTModel Size=14B, GPU Hour=1202025.05 | 87.76 | |
| Phi-4Model Size=14B, GPU Hour=∼1.0M2025.05 | 87.62 | |
| FuseChatModel Size=14B, GPU Hour=6502025.05 | 87.42 | |
| Qwen2.5-CoderModel Size=14B, GPU Hour=∼1.8M2025.05 | 79.83 | |
| LoLaTok. (B)=0.04, Cache Size=1282026.07 | 79.8 | |
| LLaMA 3.1 8BCache Size=∞2026.07 | 79.34 | |
| OURS (GDN)Tok. (B)=0.01, Cache Size=128, Param.=10.3M2026.07 | 79.34 | |
| LizardTok. (B)=0.04, Cache Size=1322026.07 | 79.3 | |
| LolCaTTok. (B)=0.04, Cache Size=642026.07 | 79.07 | |
| STILL (concurrent)Tok. (B)=0.04, Cache Size=NA2026.07 | 79 | |
| FP16BIT=16, Evaluation Protocol=Few-shot2026.07 | 78.84 | |
| OURS (GDN)Tok. (B)=0.01, Cache Size=64, Param.=10.3M2026.07 | 78.79 | |
| LlambaTok. (B)=12, Cache Size=0, Param.=8B2026.07 | 77.6 | |
| GSRQBIT=2, Evaluation Protocol=Few-shot2026.07 | 76.38 | |
| Liger-GLATok. (B)=0.02, Cache Size=642026.07 | 75.4 | |
| VQLLMBIT=2, Evaluation Protocol=Few-shot2026.07 | 75.23 | |
| GSRQBIT=1, Evaluation Protocol=Few-shot2026.07 | 69.56 | |
| VanillaBackbone=Mixtral-8x7B-v0.1, Sparsity=0%2025.11 | 64.9 | |
| PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 64.2 | |
| GSRQBIT=0.75, Evaluation Protocol=Few-shot2026.07 | 63.74 | |
| PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 63.3 | |
| Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 62 | |
| NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 61.9 | |
| HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 61.3 | |
| D2Backbone=Mixtral-8x7B-v0.1, Sparsity=20%2025.11 | 61 | |
| STUNBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.11 | 60.8 | |
| VanillaBackbone=Qwen3-MoE-30B-A3B, Sparsity=0%2025.11 | 59.5 | |
| WandaBackbone=Mixtral-8x7B-v0.1, Sparsity=2:42025.11 | 58.7 | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.11 | 58.3 | |
| VanillaBackbone=Deepseek-MoE-16b, Sparsity=0%2025.11 | 58.1 | |
| VanillaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=0%2025.11 | 58 | |
| NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 57.8 | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%2025.11 | 57.3 | |
| PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.11 | 57.2 | |
| PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.11 | 57.1 | |
| D2Backbone=Mixtral-8x7B-v0.1, Sparsity=40%2025.11 | 57 | |
| Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 57 | |
| PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2025.11 | 56.5 | |
| PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.11 | 56.3 | |
| HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.11 | 55.5 | |
| D2Backbone=Deepseek-MoE-16b, Sparsity=20%2025.11 | 55 | |
| Sub-MoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.11 | 54 | |
| WandaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=2:42025.11 | 52.6 | |
| VQLLMBIT=1, Evaluation Protocol=Few-shot2026.07 | 51.65 | |
| WandaBackbone=Deepseek-MoE-16b, Sparsity=2:42025.11 | 51.4 | |
| WandaBackbone=Qwen3-MoE-30B-A3B, Sparsity=2:42025.11 | 51.1 | |
| D2Backbone=Qwen1.5-MoE-A2.7B, Sparsity=20%2025.11 | 51 | |
| HC-SMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%2025.11 | 50.4 | |
| HC-SMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.11 | 48.1 | |
| Sub-MoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.11 | 47 | |
| PrismScale=Large, Evaluation Protocol=Zero-shot2026.06 | 46.5 | |
| BaselineScale=Large, Evaluation Protocol=Zero-shot2026.06 | 46.3 | |
| D2Backbone=Qwen3-MoE-30B-A3B, Sparsity=20%2025.11 | 45.1 | |
| D2Backbone=Deepseek-MoE-16b, Sparsity=40%2025.11 | 45 | |
| MOE+Lngram2026.05 | 44.81 | |
| MOE+Lngram-23Lconfiguration=23L2026.05 | 44.75 | |
| HC-SMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.11 | 44.6 | |
| HC-SMoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.11 | 44.3 | |
| MOE+Engram2026.05 | 44.18 | |
| Sub-MoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.11 | 44 | |
| MOE2026.05 | 43.94 | |
| D2Backbone=Qwen3-MoE-30B-A3B, Sparsity=40%2025.11 | 43.1 | |
| D2Backbone=Qwen1.5-MoE-A2.7B, Sparsity=40%2025.11 | 42.7 | |
| Sub-MoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.11 | 41 | |
| PrismScale=Medium, Evaluation Protocol=Zero-shot2026.06 | 39 | |
| BaselineScale=Medium, Evaluation Protocol=Zero-shot2026.06 | 38.7 | |
| HC-SMoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.11 | 31.2 | |
| HC-SMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2025.11 | 31.2 | |
| PrismScale=Small, Evaluation Protocol=Zero-shot2026.06 | 30 | |
| BaselineScale=Small, Evaluation Protocol=Zero-shot2026.06 | 29.6 | |
| BF16Size=1.0B2026.05 | 27.13 | |
| BF16Size=1.7B2026.05 | 27.07 | |
| BF16Size=0.4B2026.05 | 26.85 | |
| +LGRSize=0.4B2026.05 | 26.22 | |
| +LGRSize=1.7B2026.05 | 26.17 | |
| B158Size=1.7B2026.05 | 26.07 | |
| +LGRSize=1.0B2026.05 | 25.75 | |
| B158Size=1.0B2026.05 | 25.7 | |
| B158Size=0.4B2026.05 | 25.51 |