ResearchBenchmarksLanguage Modeling on AgentCompanyFollow1.247PerplexityMSA-CPT1.246921.247461.2481.24854Jun 11, 2026Evaluation ResultsMethodMethodLinksPerplexityMSA-CPTTraining Strategy=spar...Training Strategy=sparse continued pretraining2026.061.247FullTraining Strategy=Full...Training Strategy=Full-Attention baseline2026.061.248MSA-PTTraining Strategy=from...Training Strategy=from-scratch sparse pretraining2026.061.249