Language Modeling on BabyLM 2025 (test)
16.99Perplexity (PPL)SC-RFA (M2)
Evaluation Results
| Method | Links | |
|---|---|---|
| SC-RFA (M2)Evaluation context length (L)=1024, Training context window=5122025.09 | 16.99 | |
| ALiBi (B2)Evaluation context length (L)=2048, Training context window=5122025.09 | 17.06 | |
| SC-RFA (M3)Evaluation context length (L)=1024, Training context window=5122025.09 | 17.07 | |
| ALiBi (B2)Evaluation context length (L)=1024, Training context window=5122025.09 | 17.2 | |
| SC-RFA (M3)Evaluation context length (L)=2048, Training context window=5122025.09 | 17.26 | |
| SC-RFA (M2)Evaluation context length (L)=512, Training context window=5122025.09 | 17.36 | |
| ALiBi (B2)Evaluation context length (L)=4096, Training context window=5122025.09 | 17.51 | |
| RFA (M1)Evaluation context length (L)=512, Training context window=5122025.09 | 17.51 | |
| SC-RFA (M3)Evaluation context length (L)=512, Training context window=5122025.09 | 17.51 | |
| RoPE (B1)Evaluation context length (L)=512, Training context window=5122025.09 | 17.7 | |
| ALiBi (B2)Evaluation context length (L)=512, Training context window=5122025.09 | 17.7 | |
| RFA (M1)Evaluation context length (L)=1024, Training context window=5122025.09 | 17.71 | |
| SC-RFA (M2)Evaluation context length (L)=2048, Training context window=5122025.09 | 18.33 | |
| SC-RFA (M3)Evaluation context length (L)=4096, Training context window=5122025.09 | 18.74 | |
| RoPE (B1)Evaluation context length (L)=1024, Training context window=5122025.09 | 18.78 | |
| RFA (M1)Evaluation context length (L)=2048, Training context window=5122025.09 | 20.61 | |
| SC-RFA (M2)Evaluation context length (L)=4096, Training context window=5122025.09 | 22.25 | |
| RoPE (B1)Evaluation context length (L)=2048, Training context window=5122025.09 | 23.33 | |
| RFA (M1)Evaluation context length (L)=4096, Training context window=5122025.09 | 31.04 | |
| RoPE (B1)Evaluation context length (L)=4096, Training context window=5122025.09 | 33.29 |