ResearchBenchmarksLanguage Modeling on TAU2Follow1.148PerplexityMSA-PT1.147721.149611.15151.15339Jun 11, 2026Evaluation ResultsMethodMethodLinksPerplexityMSA-PTTraining Strategy=from...Training Strategy=from-scratch sparse pretraining2026.061.148MSA-CPTTraining Strategy=spar...Training Strategy=sparse continued pretraining2026.061.15FullTraining Strategy=Full...Training Strategy=Full-Attention baseline2026.061.155