Pre-training on C4 (val)
17.8PerplexityADAMW + GradientStabilizer
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ADAMW + GradientStabilizerTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 17.8 | — | — | |
| ADAM + GradientStabilizerTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 17.83 | — | — | |
| ADAM + ZCLIPTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 18.62 | — | — | |
| ADAM + AGCTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 18.76 | — | — | |
| ADAM + NORM CLIPTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 18.84 | — | — | |
| ADAMW + GradientStabilizerTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 18.84 | — | — | |
| ADAM + GradientStabilizerTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 18.89 | — | — | |
| ADAMW + ZCLIPTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 18.89 | — | — | |
| ADAMW + AGCTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 18.9 | — | — | |
| ADAMW + NORM CLIPTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 18.98 | — | — | |
| ADAM + VALUE CLIPTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 19.01 | — | — | |
| ADAMTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 19.04 | — | — | |
| ADAMW + VALUE CLIPTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 19.15 | — | — | |
| ADAMWTraining Precision=FP16, Model Size=350M, Training Tokens=6.6B2025.02 | 19.21 | — | — | |
| ADAM + ZCLIPTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 19.23 | — | — | |
| ADAM + AGCTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 19.24 | — | — | |
| ADAMW + AGCTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 19.42 | — | — | |
| ADAMW + NORM CLIPTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 20.13 | — | — | |
| ADAM + VALUE CLIPTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 20.48 | — | — | |
| ADAMWTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 21.35 | — | — | |
| ADAMW + ZCLIPTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 21.39 | — | — | |
| ADAMTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 21.57 | — | — | |
| ADAMW + VALUE CLIPTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 21.84 | — | — | |
| ADAM + NORM CLIPTraining Precision=FP4, Model Size=350M, Training Tokens=6.6B2025.02 | 22.29 | — | — | |
| ADAMW + GradientStabilizerTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 23.14 | — | — | |
| ADAM + GradientStabilizerTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 23.32 | — | — | |
| ADAMW + NORM CLIPTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 23.86 | — | — | |
| ADAMW + ZCLIPTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 23.89 | — | — | |
| ADAMW + AGCTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.09 | — | — | |
| ADAM + ZCLIPTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.16 | — | — | |
| ADAM + NORM CLIPTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.17 | — | — | |
| ADAMWTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.31 | — | — | |
| ADAM + AGCTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.32 | — | — | |
| ADAMW + VALUE CLIPTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.34 | — | — | |
| ADAM + VALUE CLIPTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.48 | — | — | |
| ADAMTraining Precision=FP16, Model Size=130M, Training Tokens=2.2B2025.02 | 24.6 | — | — | |
| Full-RankModel Size=130M, r/dmodel=256/768, Training Tokens=2.2B2024.05 | 25.08 | — | — | |
| VeLoRAModel Size=130M, r/dmodel=256/768, Training Tokens=2.2B2024.05 | 25.29 | — | — | |
| GaLoreModel Size=130M, r/dmodel=256/768, Training Tokens=2.2B2024.05 | 25.36 | — | — | |
| FLORAModel Size=130M, r/dmodel=256/768, Training Tokens=2.2B2024.05 | 25.88 | — | — | |
| ADAMW + GradientStabilizerTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 26.66 | — | — | |
| ADAM + GradientStabilizerTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 26.82 | — | — | |
| ADAMW + ZCLIPTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.04 | — | — | |
| ADAMW + NORM CLIPTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.05 | — | — | |
| ADAMW + AGCTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.26 | — | — | |
| ADAM + NORM CLIPTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.31 | — | — | |
| ADAM + ZCLIPTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.34 | — | — | |
| ADAM + AGCTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.52 | — | — | |
| ADAMWTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.72 | — | — | |
| ADAMW + VALUE CLIPTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.72 | — | — | |
| ADAM + VALUE CLIPTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 28.95 | — | — | |
| ADAMTraining Precision=FP4, Model Size=130M, Training Tokens=2.2B2025.02 | 29.02 | — | — | |
| Full-RankModel Size=60M, r/dmodel=128/256, Training Tokens=1.1B2024.05 | 33.52 | — | — | |
| VeLoRAModel Size=60M, r/dmodel=128/256, Training Tokens=1.1B2024.05 | 33.76 | — | — | |
| LORAModel Size=130M, r/dmodel=256/768, Training Tokens=2.2B2024.05 | 33.92 | — | — | |
| FLORAModel Size=60M, r/dmodel=128/256, Training Tokens=1.1B2024.05 | 34.35 | — | — | |
| GaLoreModel Size=60M, r/dmodel=128/256, Training Tokens=1.1B2024.05 | 34.88 | — | — | |
| LORAModel Size=60M, r/dmodel=128/256, Training Tokens=1.1B2024.05 | 34.99 | — | — | |
| AdamModel Scale=Llama-130M2026.04 | — | 41 | 3.21 | |
| AdamModel Scale=Llama-350M2026.04 | — | 89.31 | 3.03 | |
| GaLoreModel Scale=Llama-130M2026.04 | — | 40.73 | 3.55 | |
| GaLoreModel Scale=Llama-350M2026.04 | — | 88.22 | 3.39 | |
| LDAdamModel Scale=Llama-130M2026.04 | — | 40.73 | 3.23 | |
| LDAdamModel Scale=Llama-350M2026.04 | — | 88.22 | 2.86 | |
| OASISModel Scale=Llama-130M2026.04 | — | 37.1 | 3.28 | |
| OASISModel Scale=Llama-350M2026.04 | — | 79.46 | 3.02 |