Language Modeling on DCLM
1.863LossLlama 7B Baseline
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Llama 7B BaselineModel=Llama-3 7B, Method=Dense, Sparse steps=0, Dense steps=Full training2026.02 | 1.863 | 1 | |
| Llama 7B SparseModel=Llama-3 7B, Method=Venom Sparsity, Sparse steps=10k, Dense steps=38k2026.02 | 1.866 | 1.387 | |
| Llama 1B SparseModel=Llama-3 1B, Method=Venom Sparsity, Sparse steps=30k, Dense steps=30k2026.02 | 2.755 | 1.352 | |
| Llama 1B BaselineModel=Llama-3 1B, Method=Dense, Sparse steps=0, Dense steps=Full training2026.02 | 2.758 | 1 | |
| Sand. TransformerModel Architecture=Sandwich Transformer, Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 2.77 | — | |
| Composite: StackedModel Architecture=Composer (Stacked), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 2.77 | — | |
| Composite: StretchedModel Architecture=Composer (Stretched), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 2.77 | — | |
| Llama 3.2Model Architecture=Llama 3.2, Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 2.8 | — | |
| 1:2 Striped Attn.Model Architecture=Striped Attention (1:2 ratio), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 2.81 | — | |
| 1:4 Striped Attn.Model Architecture=Striped Attention (1:4 ratio), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 2.82 | — | |
| 1:8 Striped Attn.Model Architecture=Striped Attention (1:8 ratio), Model Parameter Scale=1B, Pre-training Dataset=DCLM, Pre-training Tokens=37.5B2025.10 | 2.85 | — |