General Downstream Evaluation on 19 short-context benchmarks
41.32ShortAvgSWA-128-NoPE
Evaluation Results
| Method | Links | |
|---|---|---|
| SWA-128-NoPEScale=S5 (0.66B), Training Tokens=≈ 100B2026.06 | 41.32 | |
| SWA-128Scale=S5 (0.66B), Training Tokens=≈ 100B2026.06 | 41.31 | |
| FullScale=S5 (0.66B), Training Tokens=≈ 100B2026.06 | 40.46 | |
| FullScale=S4 (0.22B), Training Tokens=≈ 100B2026.06 | 38.13 | |
| SWA-128Scale=S4 (0.22B), Training Tokens=≈ 100B2026.06 | 38.03 | |
| SWA-128-NoPEScale=S4 (0.22B), Training Tokens=≈ 100B2026.06 | 37.88 |