ResearchBenchmarksLanguage Modeling on FineWeb in-distributionLanguage Modeling on FineWeb in-distributionFollow19.73Perplexity (PPL)SSA19.644820.219920.79521.3701Aug 20, 2025Evaluation resultsMethodMethodLinksPerplexity (PPL)SSAModel Architecture=Nem...Model Architecture=Nemotron-style decoder, Parameters=114M, Training Steps=22K, Training Dataset=FineWeb2025.0819.73SoftmaxModel Architecture=Nem...Model Architecture=Nemotron-style decoder, Parameters=114M, Training Steps=22K, Training Dataset=FineWeb2025.0821.86