Language Modeling on (val) using Validation BPB
0.6951Validation BPBDet
Evaluation Results
| Method | Links | |
|---|---|---|
| DetVocabulary Size=128k, Depth=242026.05 | 0.6951 | |
| BiasVocabulary Size=128k, Depth=242026.05 | 0.6959 | |
| BPEVocabulary Size=128k, Depth=242026.05 | 0.6968 | |
| DetVocabulary Size=32k, Depth=242026.05 | 0.7033 | |
| BiasVocabulary Size=32k, Depth=242026.05 | 0.704 | |
| BPEVocabulary Size=32k, Depth=242026.05 | 0.7042 | |
| BPEVocabulary Size=8k, Depth=242026.05 | 0.7145 | |
| DetVocabulary Size=8k, Depth=242026.05 | 0.7154 | |
| BiasVocabulary Size=8k, Depth=242026.05 | 0.7165 | |
| Gated AttentionModel Size=2B, Training Loss=base+aux2026.02 | 0.7197 | |
| Vanilla AttentionModel Size=2B, Training Loss=base+aux2026.02 | 0.7198 | |
| Sink AttentionModel Size=2B, Training Loss=base+aux2026.02 | 0.7202 | |
| Sink AttentionModel Size=2B, Training Loss=base2026.02 | 0.7226 | |
| Gated AttentionModel Size=2B, Training Loss=base2026.02 | 0.7232 | |
| Vanilla AttentionModel Size=2B, Training Loss=base2026.02 | 0.7243 | |
| DetVocabulary Size=128k, Depth=182026.05 | 0.7504 | |
| BiasVocabulary Size=128k, Depth=182026.05 | 0.751 | |
| BPEVocabulary Size=128k, Depth=182026.05 | 0.7515 | |
| Gated AttentionModel Size=1B, Training Loss=base+aux2026.02 | 0.7547 | |
| DetVocabulary Size=32k, Depth=182026.05 | 0.756 | |
| Vanilla AttentionModel Size=1B, Training Loss=base+aux2026.02 | 0.7562 | |
| Sink AttentionModel Size=1B, Training Loss=base+aux2026.02 | 0.7567 | |
| Gated AttentionModel Size=1B, Training Loss=base2026.02 | 0.7572 | |
| Sink AttentionModel Size=1B, Training Loss=base2026.02 | 0.7583 | |
| Vanilla AttentionModel Size=1B, Training Loss=base2026.02 | 0.7591 | |
| BiasVocabulary Size=32k, Depth=182026.05 | 0.7608 | |
| BPEVocabulary Size=32k, Depth=182026.05 | 0.7616 | |
| BPEVocabulary Size=8k, Depth=182026.05 | 0.7767 | |
| DetVocabulary Size=8k, Depth=182026.05 | 0.777 | |
| BiasVocabulary Size=8k, Depth=182026.05 | 0.7793 | |
| Sink AttentionModel Size=0.6B, Training Loss=base+aux2026.02 | 0.8116 | |
| Gated AttentionModel Size=0.6B, Training Loss=base+aux2026.02 | 0.8121 | |
| Vanilla AttentionModel Size=0.6B, Training Loss=base+aux2026.02 | 0.8123 | |
| Sink AttentionModel Size=0.6B, Training Loss=base2026.02 | 0.8123 | |
| Vanilla AttentionModel Size=0.6B, Training Loss=base2026.02 | 0.8152 | |
| Gated AttentionModel Size=0.6B, Training Loss=base2026.02 | 0.8176 | |
| BiasVocabulary Size=128k, Depth=122026.05 | 0.8393 | |
| DetVocabulary Size=256k, Depth=122026.05 | 0.8398 | |
| BiasVocabulary Size=256k, Depth=122026.05 | 0.8402 | |
| DetVocabulary Size=128k, Depth=122026.05 | 0.8403 | |
| BPEVocabulary Size=128k, Depth=122026.05 | 0.841 | |
| BPEVocabulary Size=256k, Depth=122026.05 | 0.8411 | |
| DetVocabulary Size=64k, Depth=122026.05 | 0.8438 | |
| BiasVocabulary Size=64k, Depth=122026.05 | 0.8452 | |
| BPEVocabulary Size=64k, Depth=122026.05 | 0.8465 | |
| DetVocabulary Size=32k, Depth=122026.05 | 0.8525 | |
| BiasVocabulary Size=32k, Depth=122026.05 | 0.8531 | |
| BPEVocabulary Size=32k, Depth=122026.05 | 0.8536 | |
| DetVocabulary Size=16k, Depth=122026.05 | 0.8645 | |
| BPEVocabulary Size=16k, Depth=122026.05 | 0.8648 | |
| BiasVocabulary Size=16k, Depth=122026.05 | 0.8655 | |
| DetVocabulary Size=8k, Depth=122026.05 | 0.8782 | |
| BPEVocabulary Size=8k, Depth=122026.05 | 0.8785 | |
| BiasVocabulary Size=8k, Depth=122026.05 | 0.8812 | |
| FLTSize=Base, Loop=62026.05 | 0.892 | |
| FLTSize=Base, Loop=92026.05 | 0.892 | |
| FLTSize=Base, Loop=32026.05 | 0.898 | |
| LTaiSize=Base, Loop=62026.05 | 0.899 | |
| LTiSize=Base, Loop=32026.05 | 0.9 | |
| LTiSize=Base, Loop=62026.05 | 0.9 | |
| LTiSize=Base, Loop=92026.05 | 0.901 | |
| LTaiSize=Base, Loop=32026.05 | 0.901 | |
| LTaiSize=Base, Loop=92026.05 | 0.901 | |
| LTSize=Base, Loop=32026.05 | 0.905 | |
| LTSize=Base, Loop=62026.05 | 0.947 | |
| FLTSize=Small, Loop=62026.05 | 1.039 | |
| FLTSize=Small, Loop=92026.05 | 1.04 | |
| LTiSize=Small, Loop=62026.05 | 1.042 | |
| LTiSize=Small, Loop=32026.05 | 1.043 | |
| LTiSize=Small, Loop=92026.05 | 1.044 | |
| FLTSize=Small, Loop=32026.05 | 1.044 | |
| LTaiSize=Small, Loop=32026.05 | 1.046 | |
| LTaiSize=Small, Loop=92026.05 | 1.047 | |
| LTSize=Small, Loop=32026.05 | 1.054 | |
| LTSize=Small, Loop=62026.05 | 1.067 | |
| LTSize=Small, Loop=92026.05 | 1.115 |