Zero-shot Language Modeling on Wikitext and LAMBADA
25.46Wikitext PerplexityGPT-2 XLarge (Full model)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-2 XLarge (Full model)Model Layers=48, Training Strategy=from scratch, Pre-training Data=OpenWebText, Parameters=1.5B2024.04 | 25.46 | 20.24 | 22.85 | |
| InherituneModel Layers=24, Training Strategy=Inheritune, Pre-training Data=OpenWebText2024.04 | 25.52 | 16.51 | 21.01 |