LLM Pre-training Resource Utilization on C4
12.55Model Weights ShareFull params
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Full paramsBackbone=LLaMA-7B2026.03 | 12.55 | 12.55 | 25.1 | 14.66 | 64.86 | |
| GaLoreBackbone=LLaMA-7B, rank=1282026.03 | 12.55 | 12.55 | 1.73 | 4.4 | 31.23 | |
| GoLoreBackbone=LLaMA-7B, rank=1282026.03 | 12.55 | 12.55 | 1.73 | 4.4 | 31.23 | |
| LISABackbone=LLaMA-7B, sampling layers=22026.03 | 12.55 | 1.24 | 2.48 | 3.29 | 19.56 | |
| LISA-worBackbone=LLaMA-7B, sampling layers=22026.03 | 12.55 | 1.24 | 2.48 | 3.29 | 19.56 |