Downstream Task Evaluation on LAMBADA, HellaSwag, PIQA, WinoGrande, OpenBookQA, and ARC (test)
52.21HellaSwag AccuracyHiLS-Attn HoPE
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| HiLS-Attn HoPEAttention mechanism=HiLS-Attn, Positional encoding=HoPE, Model parameters=1.4B, Training tokens=300B2026.07 | 52.21 | 57.05 | 72.14 | 55.64 | 22.5 | 70.02 | 37.97 | 45.86 | 28.14 | 49.06 | |
| Full-Attn RoPEAttention mechanism=Full-Attn, Positional encoding=RoPE, Model parameters=1.4B, Training tokens=300B2026.07 | 51.99 | 57.13 | 70.51 | 56.51 | 22.6 | 68.25 | 36.95 | 44.44 | 29.49 | 48.65 | |
| Random-GuessModel parameters=1.4B, Training tokens=300B2026.07 | 25 | — | 50 | 50 | 25 | 25 | 25 | 25 | 25 | 25 |