ResearchBenchmarksLanguage Modeling on SWEFollow1.216PerplexityFull1.215921.216461.2171.21754Jun 11, 2026Evaluation ResultsMethodMethodLinksPerplexityFullTraining Strategy=Full...Training Strategy=Full-Attention baseline2026.061.216MSA-CPTTraining Strategy=spar...Training Strategy=sparse continued pretraining2026.061.216MSA-PTTraining Strategy=from...Training Strategy=from-scratch sparse pretraining2026.061.218