Language Modeling on OpenWebText 1 (val)
14.66Validation PerplexityG-Lion
Evaluation results
| Method | Links | |
|---|---|---|
| G-LionModel Configuration=GPT-2++ (760M)2024.03 | 14.66 | |
| D-Lion (MaVo)Model Configuration=GPT-2++ (760M)2024.03 | 14.66 | |
| D-Lion (Avg)Model Configuration=GPT-2++ (760M)2024.03 | 14.69 | |
| AdamWModel Configuration=GPT-2++ (760M)2024.03 | 14.7 | |
| G-LionModel Configuration=GPT-2++ (350M)2024.03 | 18.35 | |
| D-Lion (MaVo)Model Configuration=GPT-2++ (350M)2024.03 | 18.37 | |
| D-Lion (Avg)Model Configuration=GPT-2++ (350M)2024.03 | 18.39 | |
| AdamWModel Configuration=GPT-2++ (350M)2024.03 | 18.43 |