List your company
ResearchFormerly WizWand

Language Modeling on 25 M decoder

5.05Mean Final LossDelayed Nesterov
-14.252116.0365246.325376.6135May 9, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Delayed NesterovCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.055.05
SDMCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.056.46
Adam-DecayCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.056.75
CGADCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.056.79
Delayed NesterovCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.057.48
SDMCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.057.7
Adam-DecayCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.057.98
Nesterov (DiLoCo)Communication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.04
SDMCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.1
Adam-DecayCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.3
CGADCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.3
Adam-DecayCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.46
CGADCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.46
Delayed NesterovCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.48
CGADCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.058.96
SDMCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.059.46
Delayed NesterovCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.0510.16
Nesterov (DiLoCo)Communication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.0585.4
Nesterov (DiLoCo)Communication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05170.9
Nesterov (DiLoCo)Communication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05487.6