Question Answering on SciQ (Normalized Accuracy)
87.9Normalized AccuracyComposite: Stretched
Evaluation results
| Method | Links | |
|---|---|---|
| Composite: StretchedModel Architecture=Composer (Stretched), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 87.9 | |
| Composite: StackedModel Architecture=Composer (Stacked), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 87.6 | |
| STAR*Model Architecture=STAR, Model Parameter Scale=1B, Pre-training Dataset=Original Paper Dataset2025.10 | 87 | |
| Sand. TransformerModel Architecture=Sandwich Transformer, Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 83.4 | |
| Gated KalmaNetevaluation_mode=zero-shot, parameters=2.8B, context_length=< 2K tokens2025.11 | 83.2 | |
| Llama 3.2Model Architecture=Llama 3.2, Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 80.6 | |
| Gated DeltaNetevaluation_mode=zero-shot, parameters=2.8B, context_length=< 2K tokens2025.11 | 80.6 | |
| 1:2 Striped Attn.Model Architecture=Striped Attention (1:2 ratio), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 80 | |
| Mamba2evaluation_mode=zero-shot, parameters=2.8B, context_length=< 2K tokens2025.11 | 79.8 | |
| Transformerevaluation_mode=zero-shot, parameters=2.8B, context_length=< 2K tokens2025.11 | 79.5 | |
| 1:4 Striped Attn.Model Architecture=Striped Attention (1:4 ratio), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 78.3 | |
| 1:8 Striped Attn.Model Architecture=Striped Attention (1:8 ratio), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 75.7 | |
| Gated Linear Attentionevaluation_mode=zero-shot, parameters=2.8B, context_length=< 2K tokens2025.11 | 69.6 | |
| MIRParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06 | 60 | |
| Regularized BaselineParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06 | 57.8 | |
| DeltaNetevaluation_mode=zero-shot, parameters=2.8B, context_length=< 2K tokens2025.11 | 43.9 | |
| Random GuessParameters=1.4B, Training Data=DCLM, Training Tokens=100M, Evaluation=Zero-shot2026.06 | 25 |