Language Modeling on C4 Regime BM (val)
3.887Final Validation LossEchelon
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| EchelonComm. (GB)=18 ± 2, Syncs=12 ± 2, Tokens (M)=24.882026.06 | 3.887 | 48.75 | |
| DiLoCo+SAComm. (GB)=35–45, Syncs=15–25, Tokens (M)=24.882026.06 | 3.95 | 52.2 | |
| Streaming DiLoCoComm. (GB)=30–40, Syncs=30–60, Tokens (M)=24.882026.06 | 3.95 | 52.2 | |
| Standard DiLoCoComm. (GB)=30–40, Syncs=8–15, Tokens (M)=24.882026.06 | 3.95 | 52.1 | |
| Drift DiLoCoComm. (GB)=18 ± 2, Syncs=12 ± 2, Tokens (M)=24.882026.06 | 4.03 | 56 |