Science Question Answering on ARC Challenge (Accuracy %)
93AccuracyTSDS
Evaluation Results
| Method | Links | |
|---|---|---|
| TSDSIteration=Iter. 1, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 93 | |
| TSDSIteration=Iter. 1, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 92.8 | |
| Attr-DivIteration=Iter. 1, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 92.49 | |
| Attr-DivIteration=Iter. 2, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 92.15 | |
| Attr-DivIteration=Iter. 2, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.98 | |
| TSDSIteration=Iter. 2, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.98 | |
| EVOSELECTIteration=Iter. 2, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.98 | |
| EVOSELECTIteration=Iter. 2, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.98 | |
| RandomIteration=Iter. 1, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.89 | |
| EVOSELECTIteration=Iter. 1, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.89 | |
| BaseModel Scale=14b base model2026.04 | 91.81 | |
| RandomIteration=Iter. 1, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.81 | |
| AttributionIteration=Iter. 1, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.81 | |
| DiversityIteration=Iter. 1, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.81 | |
| EVOSELECTIteration=Iter. 1, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.81 | |
| RandomIteration=Iter. 2, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.81 | |
| TSDSIteration=Iter. 2, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.81 | |
| Attr-DivIteration=Iter. 1, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.72 | |
| AttributionIteration=Iter. 2, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.72 | |
| AllIteration=Iter. 1, Model Scale=14b base model2026.04 | 91.64 | |
| AllIteration=Iter. 2, Model Scale=14b base model2026.04 | 91.64 | |
| DiversityIteration=Iter. 1, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.47 | |
| RandomIteration=Iter. 2, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.47 | |
| DiversityIteration=Iter. 2, Selection Ratio=0.5, Model Scale=14b base model2026.04 | 91.47 | |
| AttributionIteration=Iter. 1, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.38 | |
| DiversityIteration=Iter. 2, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.21 | |
| AttributionIteration=Iter. 2, Selection Ratio=0.2, Model Scale=14b base model2026.04 | 91.13 | |
| MSA-CPTTraining Strategy=sparse continued pretraining2026.06 | 82.9 | |
| FullTraining Strategy=Full-Attention baseline2026.06 | 82.7 | |
| MSA-PTTraining Strategy=from-scratch sparse pretraining2026.06 | 82.5 | |
| RL-PLUSBase model=Qwen2.5-Math-7B2025.07 | 82.3 | |
| GRPOBase model=Qwen2.5-Math-7B2025.07 | 81.7 | |
| SFTBase model=Qwen2.5-Math-7B2025.07 | 75.2 | |
| SFT+GRPOBase model=Qwen2.5-Math-7B2025.07 | 72.4 | |
| GEMMA-7BState=Dense2025.06 | 49.91 | |
| DEEPSEEK-7BState=Dense2025.06 | 43.6 | |
| LLAMA-2-7BState=Dense2025.06 | 43.34 | |
| MiCRoBase Model=Llama-3.2-3B, Model architecture type=MiCRo, Few-shot=true2025.06 | 43.3 | |
| DenseBase Model=Llama-3.2-3B, Model architecture type=Dense, Few-shot=true2025.06 | 42.9 | |
| VICUNA-1.3-7BState=Dense2025.06 | 42.41 | |
| MoBBase Model=Llama-3.2-3B, Model architecture type=MoB, Few-shot=true2025.06 | 41.3 | |
| MiCRoBase Model=SmollM2-360M, Model architecture type=MiCRo, Few-shot=true2025.06 | 38.2 | |
| DenseBase Model=SmollM2-360M, Model architecture type=Dense, Few-shot=true2025.06 | 37.5 | |
| MoBBase Model=SmollM2-360M, Model architecture type=MoB, Few-shot=true2025.06 | 37.1 | |
| MASKLLMBackbone=LLAMA-2-7B2025.06 | 35.67 | |
| MASKLLMBackbone=DEEPSEEK-7B2025.06 | 35.32 | |
| DenseBase Model=Llama-3.2-1B, Model architecture type=Dense, Few-shot=true2025.06 | 33.7 | |
| MASKLLMBackbone=VICUNA-1.3-7B2025.06 | 33.57 | |
| MaskProBackbone=DEEPSEEK-7B2025.06 | 33.49 | |
| MaskProBackbone=VICUNA-1.3-7B2025.06 | 33.19 | |
| SPARSEGPTBackbone=DEEPSEEK-7B2025.06 | 32.94 | |
| MaskProBackbone=LLAMA-2-7B2025.06 | 32.85 | |
| GBLMBackbone=DEEPSEEK-7B2025.06 | 32.85 | |
| MiCRoBase Model=Llama-3.2-1B, Model architecture type=MiCRo, Few-shot=true2025.06 | 32.8 | |
| PRUNER-ZEROBackbone=VICUNA-1.3-7B2025.06 | 32.76 | |
| SPARSEGPTBackbone=VICUNA-1.3-7B2025.06 | 32.42 | |
| MoBBase Model=Llama-3.2-1B, Model architecture type=MoB, Few-shot=true2025.06 | 32.4 | |
| X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 32.2 | |
| WANDABackbone=VICUNA-1.3-7B2025.06 | 32.17 | |
| WANDABackbone=DEEPSEEK-7B2025.06 | 32 | |
| X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 31.8 | |
| SPARSEGPTBackbone=LLAMA-2-7B2025.06 | 31.48 | |
| GBLMBackbone=VICUNA-1.3-7B2025.06 | 31.4 | |
| PRUNER-ZEROBackbone=DEEPSEEK-7B2025.06 | 30.89 | |
| EngramBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 30.7 | |
| X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+1.87B, Evaluation Protocol=Zero/Few-shot2026.04 | 30.6 | |
| X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 30.5 | |
| X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 30.2 | |
| MAGNITUDEBackbone=LLAMA-2-7B2025.06 | 30.2 | |
| WANDABackbone=LLAMA-2-7B2025.06 | 30.2 | |
| MoBBase Model=SmollM2-135M, Model architecture type=MoB, Few-shot=true2025.06 | 29.9 | |
| X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 29.9 | |
| X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 29.9 | |
| GBLMBackbone=LLAMA-2-7B2025.06 | 29.86 | |
| X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 29.7 | |
| DenseBase Model=SmollM2-135M, Model architecture type=Dense, Few-shot=true2025.06 | 29.6 | |
| EngramBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 29.3 | |
| EngramBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 29 | |
| RetokenBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 28.8 | |
| MAGNITUDEBackbone=VICUNA-1.3-7B2025.06 | 28.75 | |
| BaselineBackbone Scale=1.15B Scale, Cfg=N/A, # Params=1.15B, Evaluation Protocol=Zero/Few-shot2026.04 | 28.7 | |
| MoRTBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 28.6 | |
| EngramBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 28.1 | |
| X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 28.1 | |
| X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 28 | |
| X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 27.9 | |
| MoRTBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.04 | 27.9 | |
| EngramBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 27.8 | |
| MiCRoBase Model=SmollM2-135M, Model architecture type=MiCRo, Few-shot=true2025.06 | 27.6 | |
| EngramBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 27.6 | |
| MAGNITUDEBackbone=DEEPSEEK-7B2025.06 | 27.56 | |
| X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+1.17B, Evaluation Protocol=Zero/Few-shot2026.04 | 27.5 | |
| MoRTBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 27.4 | |
| PRUNER-ZEROBackbone=LLAMA-2-7B2025.06 | 27.05 | |
| RetokenBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.04 | 27 | |
| RetokenBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.04 | 26.8 | |
| RetokenBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.04 | 26.2 | |
| MoRTBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 25.8 | |
| RetokenBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 25.5 | |
| MoRTBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.04 | 25.4 |