Long-context Understanding on LongBench 16K context
19.02LongBench Score (16k)SWA-128-NoPE
Evaluation Results
| Method | Links | |
|---|---|---|
| SWA-128-NoPEScale=S5 (0.66B), Training Tokens=≈ 100B2026.06 | 19.02 | |
| FullScale=S5 (0.66B), Training Tokens=≈ 100B2026.06 | 18.44 | |
| SWA-128Scale=S5 (0.66B), Training Tokens=≈ 100B2026.06 | 17.52 | |
| SWA-128-NoPEScale=S4 (0.22B), Training Tokens=≈ 100B2026.06 | 16.43 | |
| SWA-128Scale=S4 (0.22B), Training Tokens=≈ 100B2026.06 | 15.88 | |
| FullScale=S4 (0.22B), Training Tokens=≈ 100B2026.06 | 15.09 |