Language Model Pre-training on C4 Llama-1B scratch (val)
2.638Validation LossSF-AdamW
Evaluation results
| Method | Links | |
|---|---|---|
| SF-AdamWNumber of inner steps (H)=162025.12 | 2.638 | |
| SF-AdamWNumber of inner steps (H)=322025.12 | 2.638 | |
| SF-AdamWNumber of inner steps (H)=642025.12 | 2.638 | |
| SF-AdamWNumber of inner steps (H)=1282025.12 | 2.638 | |
| GPA-AdamWNumber of inner steps (H)=642025.12 | 2.645 | |
| GPA-AdamWNumber of inner steps (H)=1282025.12 | 2.6553 | |
| DiLoCo-AdamWNumber of inner steps (H)=1282025.12 | 2.6558 | |
| DiLoCo-AdamWNumber of inner steps (H)=642025.12 | 2.6572 | |
| DiLoCo-AdamWNumber of inner steps (H)=322025.12 | 2.6577 | |
| GPA-AdamWNumber of inner steps (H)=322025.12 | 2.6602 | |
| GPA-AdamWNumber of inner steps (H)=162025.12 | 2.6639 | |
| DiLoCo-AdamWNumber of inner steps (H)=162025.12 | 2.6737 | |
| AdamWNumber of inner steps (H)=162025.12 | 2.6749 | |
| AdamWNumber of inner steps (H)=322025.12 | 2.6749 | |
| AdamWNumber of inner steps (H)=642025.12 | 2.6749 | |
| AdamWNumber of inner steps (H)=1282025.12 | 2.6749 |