Training Stability Analysis on Llama-2 7B pre-training
0Number of SpikesAdaGC
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AdaGCTotal Steps=9K2025.02 | 0 | 0 | |
| AdaGCTotal Steps=9K2025.02 | 0 | 0 | |
| WeSaR-GlobalGCTotal Steps=9K2025.02 | 1 | 1.11 | |
| SPAMTotal Steps=9K2025.02 | 3 | 3.33 | |
| GlobalGCTotal Steps=9K2025.02 | 3 | 3.33 | |
| ClippyTotal Steps=9K2025.02 | 3 | 3.33 | |
| ScaledEmbed-GlobalGCTotal Steps=9K2025.02 | 8 | 8.89 | |
| AGCTotal Steps=9K2025.02 | 8 | 8.89 | |
| ClipByValueTotal Steps=9K2025.02 | 9 | 10 |