Science Question Answering on SciQ (Accuracy)
94.3Accuracy (SciQ)Original
Evaluation Results
| Method | Links | |
|---|---|---|
| OriginalModel=LLaMA-3.1-8B, Ratio=1.0, Evaluation Protocol=0-shot2026.04 | 94.3 | |
| mHCSize=3B, Evaluation Protocol=Five-shot2026.06 | 93.5 | |
| Depth-AttentionSize=3B, Evaluation Protocol=Five-shot2026.06 | 92.8 | |
| Attention ResidualsSize=3B, Evaluation Protocol=Five-shot2026.06 | 92.7 | |
| Attention ResidualsSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 91.9 | |
| DenseFormerSize=3B, Evaluation Protocol=Five-shot2026.06 | 91.9 | |
| mHCSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 91.8 | |
| Depth-AttentionSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 91.5 | |
| VanillaSize=3B, Evaluation Protocol=Five-shot2026.06 | 91.2 | |
| OriginalModel=LLaMA-2-7B, Ratio=1.0, Evaluation Protocol=0-shot2026.04 | 91.1 | |
| DenseFormerSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 91 | |
| VanillaSize=1.5B, Evaluation Protocol=Five-shot2026.06 | 90.1 | |
| Depth-AttentionSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 88.2 | |
| GRASPruneModel=LLaMA-3.1-8B, Ratio=0.8, Evaluation Protocol=0-shot2026.04 | 88.1 | |
| Depth-AttentionSize=3B, Evaluation Protocol=Zero-shot2026.06 | 88 | |
| FLAPModel=LLaMA-2-7B, Ratio=0.8, Evaluation Protocol=0-shot2026.04 | 87.8 | |
| DenseFormerSize=3B, Evaluation Protocol=Zero-shot2026.06 | 86.9 | |
| Attention ResidualsSize=3B, Evaluation Protocol=Zero-shot2026.06 | 86.7 | |
| mHCSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 86.6 | |
| VanillaSize=3B, Evaluation Protocol=Zero-shot2026.06 | 86.6 | |
| FLAPModel=LLaMA-3.1-8B, Ratio=0.8, Evaluation Protocol=0-shot2026.04 | 86.5 | |
| X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 86 | |
| Attention ResidualsSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 85.9 | |
| X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 85.8 | |
| mHCSize=3B, Evaluation Protocol=Zero-shot2026.06 | 85.6 | |
| DenseFormerSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 85.4 | |
| OracleModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 85.1 | |
| X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 84.8 | |
| OracleModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 84.6 | |
| X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 84.4 | |
| X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 84.2 | |
| VanillaSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 84.1 | |
| RegMix-DProxy models=128, Mode=Online2026.06 | 83.67 | |
| X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 83.5 | |
| X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 83.5 | |
| RegMix-DProxy models=512, Mode=Online2026.06 | 83.45 | |
| X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+1.87B, Evaluation Protocol=Zero/Few-shot2026.04 | 83.4 | |
| RegMix-DProxy models=128, Mode=Offline2026.06 | 83.35 | |
| X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+1.17B, Evaluation Protocol=Zero/Few-shot2026.04 | 82.9 | |
| X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 82.8 | |
| X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 82.8 | |
| MoRTBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 82.7 | |
| RegMix-DProxy models=512, Mode=Offline2026.06 | 82.53 | |
| OracleModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 82 | |
| BaselineBackbone Scale=1.15B Scale, Cfg=N/A, # Params=1.15B, Evaluation Protocol=Zero/Few-shot2026.04 | 82 | |
| RetokenBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 81.5 | |
| DoReMi2026.06 | 81.42 | |
| RetokenBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 81.2 | |
| X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 81 | |
| MoRTBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 81 | |
| RegMix2026.06 | 80.98 | |
| OracleModel=Qwen2.5-3B, Black-Box Friendly=true2026.04 | 80.8 | |
| MoRTBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 80.5 | |
| RetokenBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 80.4 | |
| OracleModel=Llama3.2-3B, Black-Box Friendly=true2026.04 | 79.8 | |
| Human2026.06 | 79.67 | |
| RetokenBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 78.8 | |
| EngramBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 78.7 | |
| RetokenBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 78.6 | |
| BaselineBackbone Scale=0.73B Scale, Cfg=N/A, # Params=0.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 78.4 | |
| MoRTBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 78.2 | |
| EngramBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 78.1 | |
| EngramBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 78 | |
| EngramBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 77.9 | |
| RetokenBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 77.5 | |
| EngramBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 77.5 | |
| MoRTBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 75 | |
| MoRTBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 74.3 | |
| RCSmedoidModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 74.2 | |
| RCSprobModel=Gemma2-9B, Black-Box Friendly=false2026.04 | 74.2 | |
| GreedyModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 74.1 | |
| GRASPruneModel=LLaMA-3.1-8B, Ratio=0.6, Evaluation Protocol=0-shot2026.04 | 74.1 | |
| RCSuniModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 74 | |
| SCModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 73.9 | |
| RCSfreqModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 73.9 | |
| EngramBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 73.9 | |
| CEModel=Gemma2-9B, Black-Box Friendly=false2026.04 | 73.6 | |
| PionArchitecture=LLaMA-1.3B, Training Tokens=54B2026.05 | 73.4 | |
| AC-ODM-410MSetting=Zero-shot, Pre-training Dataset=The Pile, Model Size=410M2025.05 | 73 | |
| NLLModel=Gemma2-9B, Black-Box Friendly=false2026.04 | 72.9 | |
| MuonArchitecture=LLaMA-1.3B, Training Tokens=54B2026.05 | 71.6 | |
| GRASPruneModel=LLaMA-2-7B, Ratio=0.8, Evaluation Protocol=0-shot2026.04 | 70.8 | |
| AdamWArchitecture=LLaMA-1.3B, Training Tokens=54B2026.05 | 70.8 | |
| GreedyModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 70.5 | |
| CEModel=Qwen2.5-7B, Black-Box Friendly=false2026.04 | 70.4 | |
| SCModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 70.3 | |
| RCSfreqModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 70.3 | |
| RCSuniModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 70.2 | |
| AC-ODMSetting=Zero-shot, Pre-training Dataset=The Pile, Model Size=1B2025.05 | 70.2 | |
| RCSmedoidModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 70.1 | |
| RCSprobModel=Qwen2.5-7B, Black-Box Friendly=false2026.04 | 70 | |
| FLAPModel=LLaMA-3.1-8B, Ratio=0.6, Evaluation Protocol=0-shot2026.04 | 70 | |
| GreedyModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 69 | |
| ODMSetting=Zero-shot, Pre-training Dataset=The Pile, Model Size=1B2025.05 | 68.9 | |
| FLAPModel=LLaMA-3.1-8B, Ratio=0.5, Evaluation Protocol=0-shot2026.04 | 68.8 | |
| RCSuniModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 68.1 | |
| RCSmedoidModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 68 | |
| RCSprobModel=Llama3.1-8B, Black-Box Friendly=false2026.04 | 67.9 | |
| KALAVAI MoEModel Scale=Pythia-6.9B, Training Steps=10000 base, Freeze Layers=6, Seed=42, Number of Examples=5002026.03 | 67.8 | |
| RCSfreqModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 67.6 |