ResearchBenchmarksLanguage Modeling on HLEFollow1.275PerplexityFull1.274881.275691.27651.27731Jun 11, 2026Evaluation ResultsMethodMethodLinksPerplexityFullTraining Strategy=Full...Training Strategy=Full-Attention baseline2026.061.275MSA-CPTTraining Strategy=spar...Training Strategy=sparse continued pretraining2026.061.275MSA-PTTraining Strategy=from...Training Strategy=from-scratch sparse pretraining2026.061.278