Downstream Language Understanding on Open LLM Leaderboard zero-shot
52.9ARCEInheritune
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| InherituneRecipe=Ours (Inheritune), Layers=16, Backbone=GPT-2 Large, Pre-training dataset=FineWeb2024.04 | 52.9 | 63.55 | 76.1 | 32.14 | 24.06 | 49.75 | |
| GPT-2 Large (rand init)Recipe=rand init, Layers=32, Backbone=GPT-2 Large, Pre-training dataset=FineWeb2024.04 | 52.48 | 64.58 | 75.3 | 32.65 | 22.2 | 49.44 | |
| InherituneRecipe=Ours (Inheritune), Layers=16, Backbone=GPT-2 Medium, Pre-training dataset=FineWeb2024.04 | 51.26 | 61.81 | 73.8 | 30.55 | 23 | 48.08 | |
| GPT-2 Medium (rand init)Recipe=rand init, Layers=24, Backbone=GPT-2 Medium, Pre-training dataset=FineWeb2024.04 | 51.05 | 61.81 | 74.8 | 30.79 | 20.28 | 47.74 | |
| GPT-2 Large (rand init)Recipe=rand init, Layers=16, Backbone=GPT-2 Large, Pre-training dataset=FineWeb2024.04 | 50.34 | 63.11 | 75 | 30.86 | 21.56 | 48.17 | |
| GPT-2 Medium (rand init)Recipe=rand init, Layers=16, Backbone=GPT-2 Medium, Pre-training dataset=FineWeb2024.04 | 49.92 | 61.92 | 73.3 | 29.56 | 19.54 | 46.84 |