Sequence Modeling on Minipile (pretraining)
17.32PerplexityLayUp
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| LayUpArchitecture=GPT-2 Medium, Evaluation Protocol=Pretraining2024.10 | 17.32 | 38,624.33 | |
| GoSGDArchitecture=GPT-2 Medium, Evaluation Protocol=Pretraining2024.10 | 17.52 | 41,147.67 | |
| SlowMoArchitecture=GPT-2 Medium, Evaluation Protocol=Pretraining2024.10 | 17.72 | 52,687 | |
| CO2Architecture=GPT-2 Medium, Evaluation Protocol=Pretraining2024.10 | 17.75 | 51,508 | |
| DDPArchitecture=GPT-2 Medium, Evaluation Protocol=Pretraining2024.10 | 17.85 | 40,658.05 | |
| AD-PSGDArchitecture=GPT-2 Medium, Evaluation Protocol=Pretraining2024.10 | 26.69 | 38,093.67 |