Language Modeling on WikiText byte-level
1.5798Wikitext PPLOLMo-2-1B-HF
Evaluation results
| Method | Links | |
|---|---|---|
| OLMo-2-1B-HFnlayers=16, dmodel=2048, rmlp/attn=3, GQA=1, Nnon-embed=1.074B2025.10 | 1.5798 | |
| LLaMA-3.2-1B-HFnlayers=16, dmodel=2048, rmlp/attn=4.8, GQA=4, Nnon-embed=973M2025.10 | 1.5807 | |
| Panda-1Bnlayers=16, dmodel=2560, rmlp/attn=1.067, GQA=4, Nnon-embed=975M2025.10 | 1.7016 | |
| Surefire-1Bnlayers=16, dmodel=2560, rmlp/attn=3.6, GQA=9, Nnon-embed=965M2025.10 | 1.7142 | |
| LLaMA-3.2-1Bnlayers=16, dmodel=2048, rmlp/attn=4.8, GQA=4, Nnon-embed=973M2025.10 | 1.7151 |