Language Model Pre-training on C4 (Eval Loss)
3.86Eval LossGrassWalk
Evaluation results
| Method | Links | |
|---|---|---|
| GrassWalkArch.=LLaMA-1B, Peak Mem. (GB)=32.0, Wall Time (m)=418.62025.10 | 3.86 | |
| GrassJumpArch.=LLaMA-1B, Peak Mem. (GB)=32.1, Wall Time (m)=415.22025.10 | 3.87 | |
| SubTrack++Arch.=LLaMA-1B, Peak Mem. (GB)=32.6, Wall Time (m)=429.22025.10 | 3.89 | |
| AdamW [Full-Rank]Arch.=LLaMA-1B, Peak Mem. (GB)=35.2, Wall Time (m)=417.02025.10 | 4.1 | |
| LDAdamArch.=LLaMA-1B, Peak Mem. (GB)=34.9, Wall Time (m)=532.82025.10 | 4.1 | |
| FRUGALArch.=LLaMA-1B, Peak Mem. (GB)=39.3, Wall Time (m)=405.12025.10 | 4.22 | |
| GrassJumpArch.=Qwen-1.5B, Peak Mem. (GB)=33.1, Wall Time (m)=432.22025.10 | 4.67 | |
| GrassWalkArch.=Qwen-1.5B, Peak Mem. (GB)=33.6, Wall Time (m)=436.62025.10 | 4.68 | |
| SubTrack++Arch.=Qwen-1.5B, Peak Mem. (GB)=33.1, Wall Time (m)=436.42025.10 | 4.7 | |
| AdamW [Full-Rank]Arch.=Qwen-1.5B, Peak Mem. (GB)=37.7, Wall Time (m)=421.02025.10 | 4.84 | |
| APOLLOArch.=LLaMA-1B, Peak Mem. (GB)=35.5, Wall Time (m)=410.52025.10 | 5.71 | |
| GaLoreArch.=LLaMA-1B, Peak Mem. (GB)=31.1, Wall Time (m)=522.22025.10 | 6.17 |