Language Modeling on LLaMA Pretraining
2.74Final LossADAMW
Evaluation Results
| Method | Links | |
|---|---|---|
| ADAMWSCALE=1B, RANK=2048, BYTES/STEP=5.09G, PEAK BYTES=5.09G, MEMORY=7.77G, UPDATE TIME=3.19s2026.02 | 2.74 | |
| GALORESCALE=1B, RANK=512, K=200, BYTES/STEP=1.48G, PEAK BYTES=3.63G, MEMORY=4.5G, UPDATE TIME=4.01s2026.02 | 2.75 | |
| ADAMWSCALE=350M, RANK=1024, BYTES/STEP=1.34G, PEAK BYTES=1.34G, MEMORY=2.08G, UPDATE TIME=0.50s2026.02 | 2.93 | |
| GALORESCALE=350M, RANK=256, K=200, BYTES/STEP=0.44G, PEAK BYTES=0.98G, MEMORY=1.26G, UPDATE TIME=0.83s2026.02 | 2.99 | |
| TSRSCALE=1B, RANK=512(256), K=100, BYTES/STEP=0.21G, PEAK BYTES=2.05G, MEMORY=3.81G, UPDATE TIME=3.19s2026.02 | 3.08 | |
| TSRSCALE=350M, RANK=384(128), K=100, BYTES/STEP=0.11G, PEAK BYTES=0.79G, MEMORY=1.19G, UPDATE TIME=0.69s2026.02 | 3.16 | |
| ADAMWSCALE=130M, RANK=768, BYTES/STEP=0.44G, PEAK BYTES=0.44G, MEMORY=0.71G, UPDATE TIME=0.41s2026.02 | 3.22 | |
| GALORESCALE=130M, RANK=256, K=200, BYTES/STEP=0.21G, PEAK BYTES=0.36G, MEMORY=0.51G, UPDATE TIME=0.59s2026.02 | 3.23 | |
| TSRSCALE=130M, RANK=384(96), K=100, BYTES/STEP=0.058G, PEAK BYTES=0.31G, MEMORY=0.45G, UPDATE TIME=0.41s2026.02 | 3.36 | |
| ADAMWSCALE=60M, RANK=512, BYTES/STEP=0.17G, PEAK BYTES=0.17G, MEMORY=0.28G, UPDATE TIME=0.42s2026.02 | 3.53 | |
| GALORESCALE=60M, RANK=128, K=200, BYTES/STEP=0.10G, PEAK BYTES=0.14G, MEMORY=0.21G, UPDATE TIME=0.43s2026.02 | 3.55 | |
| TSRSCALE=60M, RANK=256(64), K=100, BYTES/STEP=0.020G, PEAK BYTES=0.10G, MEMORY=0.17G, UPDATE TIME=0.45s2026.02 | 3.61 |