Language Modeling on Pre-training (val)
1.602Validation LossNexus
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| NexusModel Size=3B, Optimizer=Nexus2026.04 | 1.602 | — | |
| AdamWModel Size=3B, Optimizer=AdamW2026.04 | 1.606 | — | |
| AdamWModel Size=1B, Optimizer=AdamW2026.04 | 1.826 | — | |
| NexusModel Size=1B, Optimizer=Nexus2026.04 | 1.826 | — | |
| DP MuonModel Size=15B, Optimizer=DP Muon2025.05 | 1.8642 | — | |
| MuLoCo K=1Model Size=15B, Optimizer=MuLoCo, K (Communication Frequency)=12025.05 | 1.8839 | — | |
| DP AdamWModel Size=15B, Optimizer=DP AdamW2025.05 | 1.8873 | — | |
| DiLoCo K=1Model Size=15B, Optimizer=DiLoCo, K (Communication Frequency)=12025.05 | 1.8914 | — | |
| DiLoCo K=16Model Size=15B, Optimizer=DiLoCo, K (Communication Frequency)=162025.05 | 1.9062 | — | |
| MuLoCo K=16Model Size=15B, Optimizer=MuLoCo, K (Communication Frequency)=162025.05 | 1.9168 | — | |
| SATFormerVariant=XL2026.05 | 2.139 | — | |
| ResFormerVariant=XL2026.05 | 2.15 | — | |
| TransformerVariant=XL2026.05 | 2.152 | — | |
| MUDDFormerVariant=LARGE2026.05 | 2.233 | — | |
| HyperConnectionsVariant=LARGE2026.05 | 2.234 | — | |
| SATFormerVariant=LARGE2026.05 | 2.242 | — | |
| ResFormerVariant=LARGE2026.05 | 2.246 | — | |
| TransformerVariant=LARGE2026.05 | 2.256 | — | |
| DenseFormerVariant=LARGE2026.05 | 2.256 | — | |
| HyperConnectionsVariant=MEDIUM2026.05 | 2.41 | — | |
| MUDDFormerVariant=MEDIUM2026.05 | 2.415 | — | |
| SATFormerVariant=MEDIUM2026.05 | 2.415 | — | |
| ResFormerVariant=MEDIUM2026.05 | 2.429 | — | |
| DenseFormerVariant=MEDIUM2026.05 | 2.452 | — | |
| TransformerVariant=MEDIUM2026.05 | 2.461 | — | |
| SATFormerVariant=SMALL2026.05 | 2.648 | — | |
| MUDDFormerVariant=SMALL2026.05 | 2.658 | — | |
| HyperConnectionsVariant=SMALL2026.05 | 2.67 | — | |
| ResFormerVariant=SMALL2026.05 | 2.674 | — | |
| DenseFormerVariant=SMALL2026.05 | 2.704 | — | |
| TransformerVariant=SMALL2026.05 | 2.707 | — | |
| AIRAhybrid-DScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.05 | 2.719 | — | |
| AIRAhybrid-DScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stacked2026.05 | 2.72 | — | |
| Composer (2Mb-M-3A)Scale=1B, Token Budget=37.5B tokens2026.05 | 2.724 | — | |
| AIRAhybrid-BScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.05 | 2.728 | — | |
| Nemotron-2 Approx.Scale=1B, Token Budget=37.5B tokens2026.05 | 2.732 | — | |
| AIRAhybrid-BScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stacked2026.05 | 2.732 | — | |
| AIRAhybrid-EScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.05 | 2.732 | — | |
| AIRAf.-D (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.734 | — | |
| AIRAhybrid-EScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stacked2026.05 | 2.736 | — | |
| AIRAhybrid-CScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.05 | 2.74 | — | |
| Nemotron-H Approx.Scale=1B, Token Budget=37.5B tokens2026.05 | 2.741 | — | |
| AIRAf.-D (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.745 | — | |
| AIRAf.-C (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.75 | — | |
| AIRAhybrid-AScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.05 | 2.75 | — | |
| AIRAf.-C (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.751 | — | |
| AIRAf.-A (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.752 | — | |
| AIRAf.-B (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.752 | — | |
| Composite (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.759 | — | |
| Mamba (Mb + M)Scale=1B, Token Budget=37.5B tokens2026.05 | 2.771 | — | |
| Composite (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.782 | — | |
| ConventionalSize=1074M, Attention Size=269M, FFN Size=805M, dmodel=2048, dh=8192, L=16, K=1, LR=4 x 10^-4, Batch (Tokens)=4M, Tokens (B)=212026.02 | 2.81 | 20.002 | |
| HourglassSize=1074M, Attention Size=649M, FFN Size=425M, dmodel=2848, dh=2486, L=20, K=1, LR=4 x 10^-4, Batch (Tokens)=4M, Tokens (B)=212026.02 | 2.81 | 20.082 | |
| g=4, 2× widthTotal Parameters=4.71B, g=4, width=2×, Saved=33.8%2026.06 | 2.81 | 16.64 | |
| Llama 3.2Architecture=Transformer, Model Scale=1B, Token Budget=37.5B tokens2026.05 | 2.815 | — | |
| baseline (g=1)Total Parameters=7.12B, g=12026.06 | 2.82 | 16.78 | |
| g=4Total Parameters=3.50B, g=4, Saved=50.9%2026.06 | 2.82 | 16.86 | |
| Conventional (OLMo-2)Size=1074M, Pre-trained Model=OLMo2-1B2026.02 | 2.833 | 20.466 | |
| HourglassSize=906M, Attention Size=415M, FFN Size=491M, dmodel=2080, dh=819, L=24, K=4, LR=2.5 x 10^-4, Batch (Tokens)=2M, Tokens (B)=162026.02 | 2.934 | 22.282 | |
| ConventionalSize=906M, Attention Size=227M, FFN Size=679M, dmodel=1536, dh=6144, L=24, K=1, LR=2.5 x 10^-4, Batch (Tokens)=2M, Tokens (B)=162026.02 | 2.943 | 22.473 | |
| HourglassSize=403M, Attention Size=150M, FFN Size=253M, dmodel=1376, dh=557, L=24, K=4, LR=3 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=72026.02 | 3.064 | 24.909 | |
| ConventionalSize=403M, Attention Size=101M, FFN Size=302M, dmodel=1024, dh=4096, L=24, K=1, LR=3 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=72026.02 | 3.087 | 25.398 | |
| HourglassSize=113M, Attention Size=51M, FFN Size=62M, dmodel=1032, dh=418, L=12, K=4, LR=6 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=2.52026.02 | 3.426 | 35.335 | |
| ConventionalSize=113M, Attention Size=28M, FFN Size=85M, dmodel=768, dh=3072, L=12, K=1, LR=6 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=2.52026.02 | 3.464 | 36.441 | |
| COLD-INIT-1.3BInitialization=Cold, Model Scale=1.3B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05 | — | 6.2 | |
| TWIST-1.3BInitialization=Warm (TWIST), Model Scale=1.3B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05 | — | 5.93 | |
| TWIST-13BInitialization=Warm (TWIST), Model Scale=13B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05 | — | 5.34 | |
| TWIST-7BInitialization=Warm (TWIST), Model Scale=7B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05 | — | 5.47 |