Language Modeling on DCLM (val)
2.589LossIsoLoCo
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| IsoLoCoModel size=512M, Number of workers (R)=12026.07 | 2.589 | — | — | — | — | |
| IsoLoCoModel size=512M, Number of workers (R)=22026.07 | 2.593 | — | — | — | — | |
| IsoLoCoModel size=512M, Number of workers (R)=42026.07 | 2.599 | — | — | — | — | |
| IsoLoCoModel size=512M, Number of workers (R)=82026.07 | 2.61 | — | — | — | — | |
| IsoLoCoModel size=512M, Number of workers (R)=162026.07 | 2.628 | — | — | — | — | |
| DiLoCoModel size=512M, Number of workers (R)=12026.07 | 2.63 | — | — | — | — | |
| AdamW DPModel size=512M2026.07 | 2.633 | — | — | — | — | |
| DiLoCoModel size=512M, Number of workers (R)=22026.07 | 2.639 | — | — | — | — | |
| DiLoCoModel size=512M, Number of workers (R)=42026.07 | 2.653 | — | — | — | — | |
| DiLoCoModel size=512M, Number of workers (R)=82026.07 | 2.675 | — | — | — | — | |
| IsoLoCoModel size=512M, Number of workers (R)=322026.07 | 2.676 | — | — | — | — | |
| UncompressedScale=1B, Pipeline stages=P=42026.06 | 2.68 | — | 4,096 | — | — | |
| UncompressedScale=1B, Pipeline stages=P=82026.06 | 2.68 | — | 4,096 | — | — | |
| DiLoCoModel size=512M, Number of workers (R)=162026.07 | 2.706 | — | — | — | — | |
| IsoLoCoModel size=512M, Number of workers (R)=642026.07 | 2.716 | — | — | — | — | |
| MAPLScale=1B, Pipeline stages=P=42026.06 | 2.72 | 1.38 | 512 | 8 | — | |
| MAPLScale=1B, Pipeline stages=P=82026.06 | 2.73 | 2.02 | 512 | 8 | — | |
| MAPL+ VQScale=1B, Pipeline stages=P=82026.06 | 2.74 | 2.3 | 256 | 16 | — | |
| MAPL+ VQScale=1B, Pipeline stages=P=42026.06 | 2.76 | 3.01 | 256 | 16 | — | |
| DiLoCoModel size=512M, Number of workers (R)=322026.07 | 2.767 | — | — | — | — | |
| MAPLScale=500M, Pipeline stages=P=42026.06 | 2.79 | -1.9 | 512 | 6 | — | |
| UncompressedScale=500M, Pipeline stages=P=42026.06 | 2.84 | — | 3,072 | — | — | |
| UncompressedScale=500M, Pipeline stages=P=82026.06 | 2.84 | — | 3,072 | — | — | |
| MAPLScale=500M, Pipeline stages=P=82026.06 | 2.84 | 0 | 512 | 6 | — | |
| IsoLoCoModel size=178M, Number of workers (R)=12026.07 | 2.842 | — | — | — | — | |
| IsoLoCoModel size=178M, Number of workers (R)=22026.07 | 2.852 | — | — | — | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=12026.07 | 2.855 | — | — | — | — | |
| DiLoCoModel size=512M, Number of workers (R)=642026.07 | 2.863 | — | — | — | — | |
| IsoLoCoModel size=178M, Number of workers (R)=42026.07 | 2.864 | — | — | — | — | |
| DiLoCoModel size=178M, Number of workers (R)=12026.07 | 2.869 | — | — | — | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=22026.07 | 2.872 | — | — | — | — | |
| MAPL+ VQScale=500M, Pipeline stages=P=82026.06 | 2.88 | 1.49 | 256 | 12 | — | |
| AdamW DPModel size=178M2026.07 | 2.881 | — | — | — | — | |
| DiLoCoModel size=178M, Number of workers (R)=22026.07 | 2.882 | — | — | — | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=12026.07 | 2.887 | — | — | — | — | |
| IsoLoCoModel size=178M, Number of workers (R)=82026.07 | 2.889 | — | — | — | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=42026.07 | 2.891 | — | — | — | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=22026.07 | 2.905 | — | — | — | — | |
| DiLoCoModel size=178M, Number of workers (R)=42026.07 | 2.911 | — | — | — | — | |
| MAPL+ VQScale=500M, Pipeline stages=P=42026.06 | 2.92 | 2.75 | 256 | 12 | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=82026.07 | 2.929 | — | — | — | — | |
| IsoLoCoModel size=178M, Number of workers (R)=162026.07 | 2.93 | — | — | — | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=42026.07 | 2.937 | — | — | — | — | |
| DiLoCoModel size=178M, Number of workers (R)=82026.07 | 2.95 | — | — | — | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=82026.07 | 2.99 | — | — | — | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=162026.07 | 2.992 | — | — | — | — | |
| IsoLoCoModel size=178M, Number of workers (R)=322026.07 | 2.992 | — | — | — | — | |
| DiLoCoModel size=178M, Number of workers (R)=162026.07 | 3.005 | — | — | — | — | |
| SSNScale=1B, Pipeline stages=P=42026.06 | 3.05 | 13.93 | 512 | 8 | — | |
| IsoLoCoModel size=178M, Number of workers (R)=642026.07 | 3.062 | — | — | — | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=162026.07 | 3.064 | — | — | — | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=322026.07 | 3.07 | — | — | — | — | |
| DiLoCoModel size=178M, Number of workers (R)=322026.07 | 3.077 | — | — | — | — | |
| SSNScale=1B, Pipeline stages=P=82026.06 | 3.08 | 15.05 | 512 | 8 | — | |
| SSNScale=500M, Pipeline stages=P=42026.06 | 3.09 | 8.92 | 512 | 6 | — | |
| SSNScale=500M, Pipeline stages=P=82026.06 | 3.12 | 9.9 | 512 | 6 | — | |
| UncompressedScale=150M, Pipeline stages=P=42026.06 | 3.13 | — | 2,048 | — | — | |
| UncompressedScale=150M, Pipeline stages=P=82026.06 | 3.13 | — | 2,048 | — | — | |
| MAPLScale=150M, Pipeline stages=P=42026.06 | 3.156 | 0.84 | 512 | 4 | — | |
| IsoLoCoModel size=178M, Number of workers (R)=1282026.07 | 3.16 | — | — | — | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=642026.07 | 3.162 | — | — | — | — | |
| MAPLScale=150M, Pipeline stages=P=82026.06 | 3.165 | 1.11 | 512 | 4 | — | |
| MAPL+ VQScale=150M, Pipeline stages=P=42026.06 | 3.165 | 1.11 | 256 | 8 | — | |
| MAPL+ VQScale=150M, Pipeline stages=P=82026.06 | 3.17 | 1.28 | 256 | 8 | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=322026.07 | 3.178 | — | — | — | — | |
| DiLoCoModel size=178M, Number of workers (R)=642026.07 | 3.195 | — | — | — | — | |
| SSN (AdamW version)Scale=500M, Pipeline stages=P=42026.06 | 3.25 | 14.34 | 512 | 6 | — | |
| DiLoCo SGD + Iso-CModel size=178M, Number of workers (R)=1282026.07 | 3.28 | — | — | — | — | |
| SSN (AdamW version)Scale=500M, Pipeline stages=P=82026.06 | 3.29 | 15.9 | 512 | 6 | — | |
| DiLoCoModel size=178M, Number of workers (R)=1282026.07 | 3.355 | — | — | — | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=642026.07 | 3.357 | — | — | — | — | |
| SSN (AdamW version)Scale=1B, Pipeline stages=P=42026.06 | 3.38 | 26.34 | 512 | 8 | — | |
| SSNScale=150M, Pipeline stages=P=42026.06 | 3.39 | 8.37 | 512 | 4 | — | |
| SSN (AdamW version)Scale=1B, Pipeline stages=P=82026.06 | 3.39 | 26.42 | 512 | 8 | — | |
| SSNScale=150M, Pipeline stages=P=82026.06 | 3.4 | 8.63 | 512 | 4 | — | |
| SSN (AdamW version)Scale=150M, Pipeline stages=P=42026.06 | 3.49 | 11.49 | 512 | 4 | — | |
| SSN (AdamW version)Scale=150M, Pipeline stages=P=82026.06 | 3.52 | 12.46 | 512 | 4 | — | |
| DiLoCo SGDModel size=178M, Number of workers (R)=1282026.07 | 3.601 | — | — | — | — | |
| Arch-WarmupModel Parameters=1B, Model Layers=16, Architecture=Transformer2026.06 | — | — | — | — | 22.64 | |
| BaselineModel Parameters=1B, Model Layers=16, Architecture=Transformer2026.06 | — | — | — | — | 165.62 | |
| QK-NormModel Parameters=1B, Model Layers=16, Architecture=Transformer2026.06 | — | — | — | — | 61.57 | |
| SoftcapModel Parameters=1B, Model Layers=16, Architecture=Transformer2026.06 | — | — | — | — | 43.38 |