List your company
ResearchFormerly WizWand

Language Modeling on (val) using Validation BPB

0.6951Validation BPBDet
0.6783040.7916770.905051.018423Feb 1, 2026Feb 19, 2026Mar 9, 2026Mar 27, 2026Apr 14, 2026May 2, 2026May 21, 2026
Updated 5mo ago

Evaluation results

MethodLinks
DetVocabulary Size=128k, Depth=242026.050.6951
BiasVocabulary Size=128k, Depth=242026.050.6959
BPEVocabulary Size=128k, Depth=242026.050.6968
DetVocabulary Size=32k, Depth=242026.050.7033
BiasVocabulary Size=32k, Depth=242026.050.704
BPEVocabulary Size=32k, Depth=242026.050.7042
BPEVocabulary Size=8k, Depth=242026.050.7145
DetVocabulary Size=8k, Depth=242026.050.7154
BiasVocabulary Size=8k, Depth=242026.050.7165
Gated AttentionModel Size=2B, Training Loss=base+aux2026.020.7197
Vanilla AttentionModel Size=2B, Training Loss=base+aux2026.020.7198
Sink AttentionModel Size=2B, Training Loss=base+aux2026.020.7202
Sink AttentionModel Size=2B, Training Loss=base2026.020.7226
Gated AttentionModel Size=2B, Training Loss=base2026.020.7232
Vanilla AttentionModel Size=2B, Training Loss=base2026.020.7243
DetVocabulary Size=128k, Depth=182026.050.7504
BiasVocabulary Size=128k, Depth=182026.050.751
BPEVocabulary Size=128k, Depth=182026.050.7515
Gated AttentionModel Size=1B, Training Loss=base+aux2026.020.7547
DetVocabulary Size=32k, Depth=182026.050.756
Vanilla AttentionModel Size=1B, Training Loss=base+aux2026.020.7562
Sink AttentionModel Size=1B, Training Loss=base+aux2026.020.7567
Gated AttentionModel Size=1B, Training Loss=base2026.020.7572
Sink AttentionModel Size=1B, Training Loss=base2026.020.7583
Vanilla AttentionModel Size=1B, Training Loss=base2026.020.7591
BiasVocabulary Size=32k, Depth=182026.050.7608
BPEVocabulary Size=32k, Depth=182026.050.7616
BPEVocabulary Size=8k, Depth=182026.050.7767
DetVocabulary Size=8k, Depth=182026.050.777
BiasVocabulary Size=8k, Depth=182026.050.7793
Sink AttentionModel Size=0.6B, Training Loss=base+aux2026.020.8116
Gated AttentionModel Size=0.6B, Training Loss=base+aux2026.020.8121
Vanilla AttentionModel Size=0.6B, Training Loss=base+aux2026.020.8123
Sink AttentionModel Size=0.6B, Training Loss=base2026.020.8123
Vanilla AttentionModel Size=0.6B, Training Loss=base2026.020.8152
Gated AttentionModel Size=0.6B, Training Loss=base2026.020.8176
BiasVocabulary Size=128k, Depth=122026.050.8393
DetVocabulary Size=256k, Depth=122026.050.8398
BiasVocabulary Size=256k, Depth=122026.050.8402
DetVocabulary Size=128k, Depth=122026.050.8403
BPEVocabulary Size=128k, Depth=122026.050.841
BPEVocabulary Size=256k, Depth=122026.050.8411
DetVocabulary Size=64k, Depth=122026.050.8438
BiasVocabulary Size=64k, Depth=122026.050.8452
BPEVocabulary Size=64k, Depth=122026.050.8465
DetVocabulary Size=32k, Depth=122026.050.8525
BiasVocabulary Size=32k, Depth=122026.050.8531
BPEVocabulary Size=32k, Depth=122026.050.8536
DetVocabulary Size=16k, Depth=122026.050.8645
BPEVocabulary Size=16k, Depth=122026.050.8648
BiasVocabulary Size=16k, Depth=122026.050.8655
DetVocabulary Size=8k, Depth=122026.050.8782
BPEVocabulary Size=8k, Depth=122026.050.8785
BiasVocabulary Size=8k, Depth=122026.050.8812
FLTSize=Base, Loop=62026.050.892
FLTSize=Base, Loop=92026.050.892
FLTSize=Base, Loop=32026.050.898
LTaiSize=Base, Loop=62026.050.899
LTiSize=Base, Loop=32026.050.9
LTiSize=Base, Loop=62026.050.9
LTiSize=Base, Loop=92026.050.901
LTaiSize=Base, Loop=32026.050.901
LTaiSize=Base, Loop=92026.050.901
LTSize=Base, Loop=32026.050.905
LTSize=Base, Loop=62026.050.947
FLTSize=Small, Loop=62026.051.039
FLTSize=Small, Loop=92026.051.04
LTiSize=Small, Loop=62026.051.042
LTiSize=Small, Loop=32026.051.043
LTiSize=Small, Loop=92026.051.044
FLTSize=Small, Loop=32026.051.044
LTaiSize=Small, Loop=32026.051.046
LTaiSize=Small, Loop=92026.051.047
LTSize=Small, Loop=32026.051.054
LTSize=Small, Loop=62026.051.067
LTSize=Small, Loop=92026.051.115