Language Modeling on Llama decoder 10M
5.56Mean Final LossDelayed Nesterov
Evaluation Results
| Method | Links | |
|---|---|---|
| Delayed Nesterovtau=02026.05 | 5.56 | |
| SDMtau=82026.05 | 6.73 | |
| Adam outertau=02026.05 | 6.79 | |
| Adam-Decaytau=02026.05 | 6.79 | |
| CGADtau=02026.05 | 6.79 | |
| Nesterov (DiLoCo)tau=02026.05 | 7.57 | |
| Eager-Async DiLoCotau=02026.05 | 7.57 | |
| Momentum Look-Aheadtau=02026.05 | 7.57 | |
| Poly-Decaytau=02026.05 | 7.57 | |
| SDMtau=02026.05 | 7.57 | |
| CGADtau=162026.05 | 7.57 | |
| Delayed Nesterovtau=42026.05 | 7.63 | |
| SDMtau=42026.05 | 7.68 | |
| CGADtau=82026.05 | 7.84 | |
| Adam-Decaytau=42026.05 | 7.93 | |
| Adam-Decaytau=162026.05 | 7.95 | |
| CGADtau=42026.05 | 8.07 | |
| Adam-Decaytau=82026.05 | 8.13 | |
| Delayed Nesterovtau=82026.05 | 8.15 | |
| Adam outertau=42026.05 | 8.63 | |
| Poly-Decaytau=42026.05 | 8.9 | |
| Adam outertau=82026.05 | 9.78 | |
| SDMtau=162026.05 | 9.99 | |
| Poly-Decaytau=82026.05 | 11.6 | |
| Delayed Nesterovtau=162026.05 | 17.77 | |
| Adam outertau=162026.05 | 17.85 | |
| Poly-Decaytau=162026.05 | 21.33 | |
| Momentum Look-Aheadtau=42026.05 | 31.67 | |
| Nesterov (DiLoCo)tau=42026.05 | 62.05 | |
| Eager-Async DiLoCotau=42026.05 | 62.05 | |
| Nesterov (DiLoCo)tau=82026.05 | 91.04 | |
| Eager-Async DiLoCotau=82026.05 | 91.04 | |
| Momentum Look-Aheadtau=82026.05 | 127.38 | |
| Momentum Look-Aheadtau=162026.05 | 182.12 | |
| Nesterov (DiLoCo)tau=162026.05 | 325.58 | |
| Eager-Async DiLoCotau=162026.05 | 325.58 |