List your company
ResearchFormerly WizWand

Language Modeling on Llama decoder 10M

5.56Mean Final LossDelayed Nesterov
-7.240879.1646165.57251.9754May 9, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Delayed Nesterovtau=02026.055.56
SDMtau=82026.056.73
Adam outertau=02026.056.79
Adam-Decaytau=02026.056.79
CGADtau=02026.056.79
Nesterov (DiLoCo)tau=02026.057.57
Eager-Async DiLoCotau=02026.057.57
Momentum Look-Aheadtau=02026.057.57
Poly-Decaytau=02026.057.57
SDMtau=02026.057.57
CGADtau=162026.057.57
Delayed Nesterovtau=42026.057.63
SDMtau=42026.057.68
CGADtau=82026.057.84
Adam-Decaytau=42026.057.93
Adam-Decaytau=162026.057.95
CGADtau=42026.058.07
Adam-Decaytau=82026.058.13
Delayed Nesterovtau=82026.058.15
Adam outertau=42026.058.63
Poly-Decaytau=42026.058.9
Adam outertau=82026.059.78
SDMtau=162026.059.99
Poly-Decaytau=82026.0511.6
Delayed Nesterovtau=162026.0517.77
Adam outertau=162026.0517.85
Poly-Decaytau=162026.0521.33
Momentum Look-Aheadtau=42026.0531.67
Nesterov (DiLoCo)tau=42026.0562.05
Eager-Async DiLoCotau=42026.0562.05
Nesterov (DiLoCo)tau=82026.0591.04
Eager-Async DiLoCotau=82026.0591.04
Momentum Look-Aheadtau=82026.05127.38
Momentum Look-Aheadtau=162026.05182.12
Nesterov (DiLoCo)tau=162026.05325.58
Eager-Async DiLoCotau=162026.05325.58