Language Modeling on OpenWebText GPT-2 (train)
3.41LossStoSignSGD
Evaluation results
| Method | Links | ||
|---|---|---|---|
| StoSignSGDModel=GPT-2, Precision=FP82026.04 | 3.41 | 30.26 | |
| LionModel=GPT-2, Precision=FP82026.04 | 3.452 | 31.55 | |
| SignSGDModel=GPT-2, Precision=FP82026.04 | 3.539 | 34.43 |