ResearchBenchmarksLanguage Modeling on OpenWebText (dev)Language Modeling on OpenWebText (dev)Follow18.32Perplexity (Dev)GELU18.288418.501718.71518.9283Jun 30, 2026Evaluation resultsMethodMethodLinksPerplexity (Dev)GELUEpoch=2, Number of lay...Epoch=2, Number of layers=12L, Parameter-matched=true2026.0618.32NC-FFNEpoch=2, Number of lay...Epoch=2, Number of layers=12L, Parameter-matched=true2026.0618.36GELUEpoch=1, Number of lay...Epoch=1, Number of layers=12L, Parameter-matched=true2026.0619NC-FFNEpoch=1, Number of lay...Epoch=1, Number of layers=12L, Parameter-matched=true2026.0619.11