Language Modeling on 25 M decoder
5.05Mean Final LossDelayed Nesterov
Evaluation Results
| Method | Links | |
|---|---|---|
| Delayed NesterovCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 5.05 | |
| SDMCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 6.46 | |
| Adam-DecayCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 6.75 | |
| CGADCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 6.79 | |
| Delayed NesterovCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 7.48 | |
| SDMCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 7.7 | |
| Adam-DecayCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 7.98 | |
| Nesterov (DiLoCo)Communication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.04 | |
| SDMCommunication delay τ=0, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.1 | |
| Adam-DecayCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.3 | |
| CGADCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.3 | |
| Adam-DecayCommunication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.46 | |
| CGADCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.46 | |
| Delayed NesterovCommunication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.48 | |
| CGADCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 8.96 | |
| SDMCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 9.46 | |
| Delayed NesterovCommunication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 10.16 | |
| Nesterov (DiLoCo)Communication delay τ=4, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 85.4 | |
| Nesterov (DiLoCo)Communication delay τ=8, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 170.9 | |
| Nesterov (DiLoCo)Communication delay τ=16, Workers (K)=4, Inner steps (H)=8, Seeds (n)=3–52026.05 | 487.6 |