Language Modeling on GPT nano (val)
3.25Validation LossReLU2
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ReLU2Model Architecture=GPT2026.05 | 3.25 | 84.2 | |
| GELUModel Architecture=GPT2026.05 | 3.26 | 89.3 | |
| NoisyReLUModel Architecture=GPT2026.05 | 3.287 | 89.2 | |
| ReLUModel Architecture=GPT2026.05 | 3.288 | 89 | |
| SUGARBSiLUModel Architecture=GPT2026.05 | 3.291 | 89.1 |