List your company
ResearchFormerly WizWand

Language Modeling on Tiny Stories

2.5Perplexity (PPL)Angular (γ=8)
2.24.2256.258.275Oct 5, 2025
Updated 8mo ago

Evaluation results

MethodLinks
Angular (γ=8)gamma=8, Sequence Length=10242025.102.5
RACE (P=4, L=4)P=4, L=4, Sequence Length=10242025.102.6
FlashAttention2Sequence Length=10242025.102.7
RACE (P=3, L=3)P=3, L=3, Sequence Length=10242025.103.2
Linformer-128Projection Dimension=128, Sequence Length=10242025.103.7
SigmoidSequence Length=10242025.103.7
RACE (P=2, L=2)P=2, L=2, Sequence Length=10242025.104.2
LinearSequence Length=10242025.107
Performer-256Dimension=256, Sequence Length=10242025.1010