Language Modeling on C4 Llama-160M scratch (val)
3.0908Validation LossGPA-AdamW
Evaluation Results
| Method | Links | |
|---|---|---|
| GPA-AdamWNumber of inner steps (H)=322025.12 | 3.0908 | |
| GPA-AdamWNumber of inner steps (H)=1282025.12 | 3.0951 | |
| GPA-AdamWNumber of inner steps (H)=642025.12 | 3.0974 | |
| SF-AdamWNumber of inner steps (H)=642025.12 | 3.1051 | |
| DiLoCo-AdamWNumber of inner steps (H)=642025.12 | 3.1051 | |
| DiLoCo-AdamWNumber of inner steps (H)=322025.12 | 3.1066 | |
| SF-AdamWNumber of inner steps (H)=82025.12 | 3.1089 | |
| SF-AdamWNumber of inner steps (H)=162025.12 | 3.1089 | |
| SF-AdamWNumber of inner steps (H)=322025.12 | 3.1089 | |
| SF-AdamWNumber of inner steps (H)=1282025.12 | 3.1089 | |
| GPA-AdamWNumber of inner steps (H)=162025.12 | 3.1089 | |
| DiLoCo-AdamWNumber of inner steps (H)=162025.12 | 3.1133 | |
| AdamWNumber of inner steps (H)=82025.12 | 3.1141 | |
| AdamWNumber of inner steps (H)=162025.12 | 3.1141 | |
| AdamWNumber of inner steps (H)=322025.12 | 3.1141 | |
| AdamWNumber of inner steps (H)=642025.12 | 3.1141 | |
| AdamWNumber of inner steps (H)=1282025.12 | 3.1141 | |
| DiLoCo-AdamWNumber of inner steps (H)=1282025.12 | 3.128 | |
| GPA-AdamWNumber of inner steps (H)=82025.12 | 3.1353 | |
| DiLoCo-AdamWNumber of inner steps (H)=82025.12 | 3.1395 |