LLM Pre-training on GPT-2 (val)
3.405LossStoSignSGDv2
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| StoSignSGDv2Precision=FP82026.04 | 3.405 | 30.12 | |
| StoSignSGDPrecision=FP82026.04 | 3.418 | 30.51 | |
| LionPrecision=FP82026.04 | 3.46 | 31.82 | |
| SignSGDPrecision=FP82026.04 | 3.546 | 34.67 |
Loading the SOTA2 catalog…