Fine-tuning on FineWeb (test)
1.93Test LossSISA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SISAModel=GPT2(335M), Epoch=152025.02 | 1.93 | 84.84 | |
| AdamWModel=GPT2(335M), Epoch=152025.02 | 1.95 | 84.19 | |
| SISAModel=GPT2(335M), Epoch=52025.02 | 2.04 | 85.24 | |
| SISAModel=GPT2(124M), Epoch=152025.02 | 2.17 | 84.82 | |
| AdamWModel=GPT2(124M), Epoch=152025.02 | 2.3 | 84.73 | |
| SISAModel=GPT2(335M), Epoch=22025.02 | 2.31 | 84.76 | |
| SISAModel=GPT2(124M), Epoch=52025.02 | 2.73 | 84.36 | |
| AdamWModel=GPT2(335M), Epoch=52025.02 | 2.73 | 84.5 | |
| AdamWModel=GPT2(124M), Epoch=52025.02 | 3.45 | 84.3 | |
| AdamWModel=GPT2(335M), Epoch=22025.02 | 3.62 | 83.17 | |
| SISAModel=GPT2(124M), Epoch=22025.02 | 3.69 | 83.15 | |
| AdamWModel=GPT2(124M), Epoch=22025.02 | 5.2 | 82.16 |