List your company
ResearchFormerly WizWand

Language Modeling on BabyLM 2025 (test)

16.99Perplexity (PPL)SC-RFA (M2)
16.33820.73925.1429.541Sep 4, 2025
Updated 6mo ago

Evaluation results

MethodLinks
SC-RFA (M2)Evaluation context length (L)=1024, Training context window=5122025.0916.99
ALiBi (B2)Evaluation context length (L)=2048, Training context window=5122025.0917.06
SC-RFA (M3)Evaluation context length (L)=1024, Training context window=5122025.0917.07
ALiBi (B2)Evaluation context length (L)=1024, Training context window=5122025.0917.2
SC-RFA (M3)Evaluation context length (L)=2048, Training context window=5122025.0917.26
SC-RFA (M2)Evaluation context length (L)=512, Training context window=5122025.0917.36
ALiBi (B2)Evaluation context length (L)=4096, Training context window=5122025.0917.51
RFA (M1)Evaluation context length (L)=512, Training context window=5122025.0917.51
SC-RFA (M3)Evaluation context length (L)=512, Training context window=5122025.0917.51
RoPE (B1)Evaluation context length (L)=512, Training context window=5122025.0917.7
ALiBi (B2)Evaluation context length (L)=512, Training context window=5122025.0917.7
RFA (M1)Evaluation context length (L)=1024, Training context window=5122025.0917.71
SC-RFA (M2)Evaluation context length (L)=2048, Training context window=5122025.0918.33
SC-RFA (M3)Evaluation context length (L)=4096, Training context window=5122025.0918.74
RoPE (B1)Evaluation context length (L)=1024, Training context window=5122025.0918.78
RFA (M1)Evaluation context length (L)=2048, Training context window=5122025.0920.61
SC-RFA (M2)Evaluation context length (L)=4096, Training context window=5122025.0922.25
RoPE (B1)Evaluation context length (L)=2048, Training context window=5122025.0923.33
RFA (M1)Evaluation context length (L)=4096, Training context window=5122025.0931.04
RoPE (B1)Evaluation context length (L)=4096, Training context window=5122025.0933.29