Reading Comprehension on RACE (Accuracy)
68.3AccuracyMSRS
Evaluation results
| Method | Links | |
|---|---|---|
| MSRSBackbone=Llama-3-8B-Instruct2025.08 | 68.3 | |
| MSRSBackbone=Mistral-7B-v0.32025.08 | 68.1 | |
| VanillaBackbone=Mistral-7B-v0.32025.08 | 67.8 | |
| VanillaBackbone=Llama-3-8B-Instruct2025.08 | 67.1 | |
| MSRSBackbone=Qwen2-7B-Instruct2025.08 | 64.8 | |
| VanillaBackbone=Qwen2-7B-Instruct2025.08 | 62.5 | |
| Dream-v0-7BType=Diffusion2026.06 | 44.7 | |
| DreamReasoner-8BType=Block Diffusion2026.06 | 43.8 | |
| Qwen3-8BType=AR2026.06 | 42.3 | |
| VICUNA-1.3-7BState=Dense2025.06 | 41.91 | |
| Qwen2.5-7BType=AR2026.06 | 41.9 | |
| GEMMA-7BState=Dense2025.06 | 40.19 | |
| OLMo3-7BType=AR2026.06 | 40.1 | |
| LLAMA-2-7BBase Model=LLaMA-2-7B, Sparsity Pattern=Dense, Evaluation Protocol=Zero-shot2025.06 | 39.62 | |
| DEEPSEEK-7BBase Model=DeepSeek-7B, Sparsity Pattern=Dense, Evaluation Protocol=Zero-shot2025.06 | 39.62 | |
| LLAMA-2-7BState=Dense2025.06 | 39.62 | |
| DEEPSEEK-7BState=Dense2025.06 | 39.62 | |
| MaskProBase Model=LLaMA-2-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 39.33 | |
| MaskProBase Model=DeepSeek-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 39.25 | |
| MASKLLMBackbone=DEEPSEEK-7B2025.06 | 39.14 | |
| MASKLLMBackbone=VICUNA-1.3-7B2025.06 | 39.13 | |
| GBLMBase Model=DeepSeek-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 38.76 | |
| MaskProBackbone=VICUNA-1.3-7B2025.06 | 38.76 | |
| LLaDA-8BType=Diffusion2026.06 | 38.7 | |
| MiMo-7BType=AR2026.06 | 38.6 | |
| MASKLLMBackbone=LLAMA-2-7B2025.06 | 38.56 | |
| SPARSEGPTBase Model=LLaMA-2-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 38.55 | |
| WANDABase Model=DeepSeek-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 38.47 | |
| GBLMBackbone=VICUNA-1.3-7B2025.06 | 38.37 | |
| PRUNER-ZEROBase Model=DeepSeek-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 38.32 | |
| WANDABase Model=LLaMA-2-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 38.18 | |
| PRUNER-ZEROBase Model=LLaMA-2-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 38.13 | |
| SPARSEGPTBase Model=DeepSeek-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 38.09 | |
| WANDABackbone=VICUNA-1.3-7B2025.06 | 37.89 | |
| SPARSEGPTBackbone=VICUNA-1.3-7B2025.06 | 37.81 | |
| SPARSEGPTBackbone=DEEPSEEK-7B2025.06 | 37.8 | |
| MaskProBackbone=DEEPSEEK-7B2025.06 | 37.75 | |
| GBLMBase Model=LLaMA-2-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 37.51 | |
| PRUNER-ZEROBackbone=VICUNA-1.3-7B2025.06 | 37.42 | |
| MaskProBackbone=LLAMA-2-7B2025.06 | 37.13 | |
| SPARSEGPTBackbone=LLAMA-2-7B2025.06 | 36.56 | |
| GBLMBackbone=DEEPSEEK-7B2025.06 | 36.17 | |
| Memory Grafting# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 47 (top-4)2026.05 | 35.98 | |
| WANDABackbone=LLAMA-2-7B2025.06 | 35.89 | |
| Depth-AttentionSize=3B, Evaluation Protocol=Five-shot2026.06 | 35.6 | |
| MAGNITUDEBase Model=LLaMA-2-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 35.31 | |
| PRUNER-ZEROBackbone=DEEPSEEK-7B2025.06 | 35.22 | |
| WANDABackbone=DEEPSEEK-7B2025.06 | 35.12 | |
| Depth-AttentionSize=3B, Evaluation Protocol=Zero-shot2026.06 | 35.02 | |
| Attention ResidualsSize=3B, Evaluation Protocol=Five-shot2026.06 | 34.83 | |
| MoE Baseline# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 64 (top-4)2026.05 | 34.74 | |
| PRUNER-ZEROBackbone=LLAMA-2-7B2025.06 | 34.64 | |
| GBLMBackbone=LLAMA-2-7B2025.06 | 34.61 | |
| Vanilla Engram# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 48 (top-4)2026.05 | 34.55 | |
| DenseFormerSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 34.45 | |
| DenseFormerSize=3B, Evaluation Protocol=Five-shot2026.06 | 34.45 | |
| DenseFormerSize=3B, Evaluation Protocol=Zero-shot2026.06 | 34.35 | |
| Depth-AttentionSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 34.07 | |
| mHCSize=3B, Evaluation Protocol=Zero-shot2026.06 | 33.97 | |
| DenseFormerSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 33.78 | |
| VanillaSize=3B, Evaluation Protocol=Zero-shot2026.06 | 33.78 | |
| mHCSize=3B, Evaluation Protocol=Five-shot2026.06 | 33.78 | |
| mHCSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 33.59 | |
| Attention ResidualsSize=3B, Evaluation Protocol=Zero-shot2026.06 | 33.49 | |
| VanillaSize=3B, Evaluation Protocol=Five-shot2026.06 | 33.49 | |
| VanillaSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 33.11 | |
| mHCSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 33.01 | |
| Attention ResidualsSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 33.01 | |
| Depth-AttentionSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 32.92 | |
| MAGNITUDEBase Model=DeepSeek-7B, Sparsity Pattern=(4:8)-sparsity, Evaluation Protocol=Zero-shot2025.06 | 32.06 | |
| Attention ResidualsSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 32.06 | |
| VanillaSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 31.96 | |
| LoopMDMS=12, Zero-shot=true2026.05 | 31.7 | |
| MAGNITUDEBackbone=LLAMA-2-7B2025.06 | 31.48 | |
| LoopMDMS=6, Zero-shot=true2026.05 | 31.3 | |
| MAGNITUDEBackbone=VICUNA-1.3-7B2025.06 | 28.61 | |
| MAGNITUDEBackbone=DEEPSEEK-7B2025.06 | 28.52 | |
| MDMZero-shot=true2026.05 | 28.5 | |
| LoopMDMS=1, Zero-shot=true2026.05 | 27.2 | |
| MaskProBackbone=GEMMA-7B2025.06 | 23.26 | |
| WANDABackbone=GEMMA-7B2025.06 | 22.78 | |
| GBLMBackbone=GEMMA-7B2025.06 | 22.49 | |
| SPARSEGPTBackbone=GEMMA-7B2025.06 | 22.39 | |
| PRUNER-ZEROBackbone=GEMMA-7B2025.06 | 21.63 | |
| MAGNITUDEBackbone=GEMMA-7B2025.06 | 21.24 | |
| MASKLLMBackbone=GEMMA-7B2025.06 | 20.1 |