ResearchBenchmarksLanguage Modeling on LLaMa-2 1B (val)Language Modeling on LLaMa-2 1B (val)Follow2.421Validation LossIsoLoCo2.419842.427672.43552.44333Jul 3, 2026Evaluation resultsMethodMethodLinksValidation LossIsoLoCoModel size=1B, Number...Model size=1B, Number of workers (R)=8, Synchronization interval (H)=302026.072.421DiLoCoModel size=1B, Number...Model size=1B, Number of workers (R)=8, Synchronization interval (H)=302026.072.45