Next-token prediction on OpenWebText (val)
18.8PerplexityHermite activation
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Hermite activationAct.=Hermite, Deg.=3, FLOPs=87.56G2025.02 | 18.8 | 2.9337 | |
| Tropical activationAct.=Tropical, Deg.=6, FLOPs=87.75G2025.02 | 18.93 | 2.941 | |
| Fourier activationAct.=Fourier, Deg.=6, FLOPs=88.69G2025.02 | 18.94 | 2.9411 | |
| GELUAct.=GELU, Deg.=null, FLOPs=87.52G2025.02 | 19.25 | 2.9574 | |
| SiLUAct.=SiLU, Deg.=null, FLOPs=87.37G2025.02 | 19.64 | 2.9775 |