List your company
ResearchFormerly WizWand

Language Modeling on Pre-training (val)

1.602Validation LossNexus
1.527522.030262.5333.03574May 29, 2025Jul 31, 2025Oct 3, 2025Dec 6, 2025Feb 7, 2026Apr 12, 2026Jun 15, 2026
Updated 4mo ago

Evaluation results

MethodLinks
NexusModel Size=3B, Optimizer=Nexus2026.041.602—
AdamWModel Size=3B, Optimizer=AdamW2026.041.606—
AdamWModel Size=1B, Optimizer=AdamW2026.041.826—
NexusModel Size=1B, Optimizer=Nexus2026.041.826—
DP MuonModel Size=15B, Optimizer=DP Muon2025.051.8642—
MuLoCo K=1Model Size=15B, Optimizer=MuLoCo, K (Communication Frequency)=12025.051.8839—
DP AdamWModel Size=15B, Optimizer=DP AdamW2025.051.8873—
DiLoCo K=1Model Size=15B, Optimizer=DiLoCo, K (Communication Frequency)=12025.051.8914—
DiLoCo K=16Model Size=15B, Optimizer=DiLoCo, K (Communication Frequency)=162025.051.9062—
MuLoCo K=16Model Size=15B, Optimizer=MuLoCo, K (Communication Frequency)=162025.051.9168—
SATFormerVariant=XL2026.052.139—
ResFormerVariant=XL2026.052.15—
TransformerVariant=XL2026.052.152—
MUDDFormerVariant=LARGE2026.052.233—
HyperConnectionsVariant=LARGE2026.052.234—
SATFormerVariant=LARGE2026.052.242—
ResFormerVariant=LARGE2026.052.246—
TransformerVariant=LARGE2026.052.256—
DenseFormerVariant=LARGE2026.052.256—
HyperConnectionsVariant=MEDIUM2026.052.41—
MUDDFormerVariant=MEDIUM2026.052.415—
SATFormerVariant=MEDIUM2026.052.415—
ResFormerVariant=MEDIUM2026.052.429—
DenseFormerVariant=MEDIUM2026.052.452—
TransformerVariant=MEDIUM2026.052.461—
SATFormerVariant=SMALL2026.052.648—
MUDDFormerVariant=SMALL2026.052.658—
HyperConnectionsVariant=SMALL2026.052.67—
ResFormerVariant=SMALL2026.052.674—
DenseFormerVariant=SMALL2026.052.704—
TransformerVariant=SMALL2026.052.707—
AIRAhybrid-DScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.052.719—
AIRAhybrid-DScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stacked2026.052.72—
Composer (2Mb-M-3A)Scale=1B, Token Budget=37.5B tokens2026.052.724—
AIRAhybrid-BScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.052.728—
Nemotron-2 Approx.Scale=1B, Token Budget=37.5B tokens2026.052.732—
AIRAhybrid-BScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stacked2026.052.732—
AIRAhybrid-EScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.052.732—
AIRAf.-D (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.052.734—
AIRAhybrid-EScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stacked2026.052.736—
AIRAhybrid-CScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.052.74—
Nemotron-H Approx.Scale=1B, Token Budget=37.5B tokens2026.052.741—
AIRAf.-D (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.052.745—
AIRAf.-C (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.052.75—
AIRAhybrid-AScale=1B, Token Budget=37.5B tokens, Architecture Variant=Stretched2026.052.75—
AIRAf.-C (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.052.751—
AIRAf.-A (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.052.752—
AIRAf.-B (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.052.752—
Composite (Str.)Architecture=Stretched, Model Scale=1B, Token Budget=37.5B tokens2026.052.759—
Mamba (Mb + M)Scale=1B, Token Budget=37.5B tokens2026.052.771—
Composite (St.)Architecture=Stacked, Model Scale=1B, Token Budget=37.5B tokens2026.052.782—
ConventionalSize=1074M, Attention Size=269M, FFN Size=805M, dmodel=2048, dh=8192, L=16, K=1, LR=4 x 10^-4, Batch (Tokens)=4M, Tokens (B)=212026.022.8120.002
HourglassSize=1074M, Attention Size=649M, FFN Size=425M, dmodel=2848, dh=2486, L=20, K=1, LR=4 x 10^-4, Batch (Tokens)=4M, Tokens (B)=212026.022.8120.082
g=4, 2× widthTotal Parameters=4.71B, g=4, width=2×, Saved=33.8%2026.062.8116.64
Llama 3.2Architecture=Transformer, Model Scale=1B, Token Budget=37.5B tokens2026.052.815—
baseline (g=1)Total Parameters=7.12B, g=12026.062.8216.78
g=4Total Parameters=3.50B, g=4, Saved=50.9%2026.062.8216.86
Conventional (OLMo-2)Size=1074M, Pre-trained Model=OLMo2-1B2026.022.83320.466
HourglassSize=906M, Attention Size=415M, FFN Size=491M, dmodel=2080, dh=819, L=24, K=4, LR=2.5 x 10^-4, Batch (Tokens)=2M, Tokens (B)=162026.022.93422.282
ConventionalSize=906M, Attention Size=227M, FFN Size=679M, dmodel=1536, dh=6144, L=24, K=1, LR=2.5 x 10^-4, Batch (Tokens)=2M, Tokens (B)=162026.022.94322.473
HourglassSize=403M, Attention Size=150M, FFN Size=253M, dmodel=1376, dh=557, L=24, K=4, LR=3 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=72026.023.06424.909
ConventionalSize=403M, Attention Size=101M, FFN Size=302M, dmodel=1024, dh=4096, L=24, K=1, LR=3 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=72026.023.08725.398
HourglassSize=113M, Attention Size=51M, FFN Size=62M, dmodel=1032, dh=418, L=12, K=4, LR=6 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=2.52026.023.42635.335
ConventionalSize=113M, Attention Size=28M, FFN Size=85M, dmodel=768, dh=3072, L=12, K=1, LR=6 x 10^-4, Batch (Tokens)=0.5M, Tokens (B)=2.52026.023.46436.441
COLD-INIT-1.3BInitialization=Cold, Model Scale=1.3B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05—6.2
TWIST-1.3BInitialization=Warm (TWIST), Model Scale=1.3B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05—5.93
TWIST-13BInitialization=Warm (TWIST), Model Scale=13B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05—5.34
TWIST-7BInitialization=Warm (TWIST), Model Scale=7B, Tokenizer Frequency=25Hz, Number of Tokens=5002023.05—5.47