Language Modeling on LLaMA-2 13B (val)
17.85Perplexity8-bit Adam
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| 8-bit AdamSteps=40K2025.09 | 17.85 | — | — | — | — | |
| CR-Net w. re-computationSteps=40K2025.09 | 18.12 | — | — | — | — | |
| CR-NetRe-computation=true, rank (r)=12602025.09 | — | 25.99 | 21.73 | 19.32 | 18.12 | |
| Full-rankOptimizer=8-bit Adam2025.09 | — | 24.31 | 20.53 | 18.84 | 17.85 |