Next-token prediction on OpenWebText (train)
18.4PPLHermite activation
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Hermite activationAct.=Hermite, Deg.=3, FLOPs=87.56G2025.02 | 18.4 | 2.9122 | |
| Fourier activationAct.=Fourier, Deg.=6, FLOPs=88.69G2025.02 | 18.85 | 2.9365 | |
| Tropical activationAct.=Tropical, Deg.=6, FLOPs=87.75G2025.02 | 18.95 | 2.9418 | |
| GELUAct.=GELU, Deg.=null, FLOPs=87.52G2025.02 | 19.05 | 2.9471 | |
| SiLUAct.=SiLU, Deg.=null, FLOPs=87.37G2025.02 | 19.46 | 2.9684 |