Language Modeling on LLaMA-2 7B pre-training (val)
16.01Validation Perplexity (40K steps)CR-Net w. re-computation
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| CR-Net w. re-computationMemory (GB)=27.602025.09 | 16.01 | 23.11 | 14.47 | 13.72 | |
| CoLA-MMemory (GB)=28.822025.09 | 16.21 | 22.76 | 14.59 | 13.82 | |
| Apollo2025.09 | 17.55 | — | — | 14.39 | |
| 8-bit GaLoreMemory (GB)=65.162025.09 | 17.94 | 26.87 | — | 15.39 | |
| 8-bit AdamMemory (GB)=72.592025.09 | 18.09 | — | — | 15.47 |