Language Modeling on Language Modeling (test)
33.19PPL (64)HSA-Ultralong
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HSA-UltralongExtra #Param=12.7%2026.07 | 33.19 | — | — | — | — | 26.26 | 21.5 | 8.81 | 5.84 | 4.99 | 4.54 | |
| Dash-Attention-RoPEExtra #Param=0.006%2026.07 | 33.7 | — | — | — | — | 26.74 | 18.67 | 5 | 10 | — | — | |
| InfLLM v2*Extra #Param=0.006%, Head Dimension=128, Note=Reported for reference2026.07 | 33.71 | — | — | — | — | 26.74 | 18.55 | 4.95 | 13.68 | 64.24 | — | |
| Full-Attn RoPEExtra #Param=–2026.07 | 33.92 | — | — | — | — | 26.89 | 18.68 | 4.96 | 10 | — | — | |
| HiLS-Attn-HoPEExtra #Param=0.6%2026.07 | 33.97 | — | — | — | — | 26.91 | 18.65 | 4.94 | 4.34 | 4.71 | 5.95 | |
| Full-Attn HoPEExtra #Param=–2026.07 | 34.15 | — | — | — | — | 26.97 | 18.73 | 4.95 | 6.42 | 10 | — | |
| NSA-RoPEExtra #Param=0.2%2026.07 | 34.15 | — | — | — | — | 27.11 | 18.85 | 5.01 | 7.62 | 11.75 | 19.14 | |
| Full-Attn HoPE (extra MLP)Extra #Param=0.6%2026.07 | 34.28 | — | — | — | — | 27.12 | 18.76 | 4.95 | 5.85 | 10 | — | |
| SWA-RoPEExtra #Param=–2026.07 | 35.38 | — | — | — | — | 27.94 | 19.3 | 8.95 | 9.01 | 8.44 | 8.47 | |
| Naive-BSA-HoPEExtra #Param=–2026.07 | 36.12 | — | — | — | — | 29.06 | 20.45 | 4.94 | 3.94 | 8.67 | — | |
| Qwen3-4B + EAGLE-2 (Group A)Backbone=Qwen3-4B, Method=EAGLE-22026.01 | — | 6.3 | 555 | 0.74 | 30 | — | — | — | — | — | — | |
| Qwen3-4B + FBS-Full (ours)Backbone=Qwen3-4B, Modules=PAW + CH + SG2026.01 | — | 6.2 | 532 | 0.7 | 36 | — | — | — | — | — | — | |
| Qwen3-4B + FBS-S1Backbone=Qwen3-4B, Modules=PAW + CH2026.01 | — | 6.3 | 755 | 1.03 | 0 | — | — | — | — | — | — | |
| Qwen3-4B + Lookahead (Group A)Backbone=Qwen3-4B, Method=Lookahead Decoding2026.01 | — | 6.4 | 595 | 0.82 | 15 | — | — | — | — | — | — | |
| Qwen3-4B + Medusa (Group A)Backbone=Qwen3-4B, Method=Medusa2026.01 | — | 6.5 | 570 | 0.8 | 18 | — | — | — | — | — | — | |
| Qwen3-4B + SpecDec (Group A)Backbone=Qwen3-4B, Method=Speculative Decoding2026.01 | — | 6.4 | 646 | 0.9 | 22 | — | — | — | — | — | — | |
| Qwen3-4B-Instruct (Baseline)Backbone=Qwen3-4B, Variant=Instruct2026.01 | — | 6.4 | 760 | 1 | 0 | — | — | — | — | — | — |