Long-context reasoning on RULER (Context Length Sweep 8K-64K)
75.3RULER Performance (8K Context)HyLo-Llama-14MLA14M2
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| HyLo-Llama-14MLA14M2Teacher=8B, KV cache=4.7%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 75.3 | 49.7 | 16.6 | 0.4 | |
| HyLo-Qwen-14MLA14M2Teacher=8B, KV cache=7.8%, Training Context Length=8K2026.04 | 74.2 | 58.6 | 33.5 | 10.7 | |
| HyLo-Qwen-14MLA14M2Teacher=8B, KV cache=7.8%, Training Context Length=64K2026.04 | 73.9 | 62.6 | — | 33.1 | |
| HyLo-Llama-14MLA14GDNTeacher=8B, KV cache=4.7%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 73.2 | 69.7 | 62.9 | 52 | |
| HyLo-Llama-14MLA14M2Teacher=8B, KV cache=4.7%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 71.7 | 65.4 | — | 46.6 | |
| Jet Nemotron-2BTeacher=-, KV cache=2.1%2026.04 | 71.3 | 60.1 | 43.9 | 14.1 | |
| HyLo-Llama-6MLA22GDNTeacher=8B, KV cache=2.0%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 71.2 | 45 | 27.1 | 14.1 | |
| HyLo-Qwen-14MLA14GDNTeacher=8B, KV cache=7.8%, Training Context Length=8K2026.04 | 71.1 | 45.6 | 19.2 | 0.2 | |
| HyLo-Llama-14MLA14GDNTeacher=8B, KV cache=4.7%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 71.1 | 45.6 | 19.2 | 0.2 | |
| HyLo-Llama-6MLA22GDNTeacher=8B, KV cache=2.0%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 68.2 | 62.1 | 55.7 | 46.3 | |
| HyLo-Qwen-14MLA14GDNTeacher=8B, KV cache=7.8%, Training Context Length=64K2026.04 | 66.9 | 53.2 | 41.4 | 31.6 | |
| HyLo-Llama-6MLA22M2Teacher=8B, KV cache=2.0%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 65.7 | 39.5 | 25.2 | 11.4 | |
| HyLo-Llama-6MLA22M2Teacher=8B, KV cache=2.0%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 65.4 | 56.4 | 49.9 | 42.3 | |
| HyLo-Qwen-7MLA21GDNTeacher=8B, KV cache=3.9%, Training Context Length=8K2026.04 | 63.5 | 43.6 | 30.3 | 17.4 | |
| M1KV cache=21.4%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 63.5 | 43.6 | 30.3 | 17.4 | |
| HyLo-Llama-8MLA8GDNTeacher=8B, KV cache=7.8%, Training Context Length=64K, Backbone Model=Llama-3.2-1B2026.04 | 61.5 | 53.7 | 48.1 | 41.6 | |
| HyLo-Llama-8MLA8GDNTeacher=8B, KV cache=7.8%, Training Context Length=8K, Backbone Model=Llama-3.2-1B2026.04 | 60.3 | 0.5 | 0.1 | 0.1 | |
| HyLo-Qwen-7MLA21GDNTeacher=8B, KV cache=3.9%, Training Context Length=64K2026.04 | 59.8 | 53.8 | 42.5 | 30.5 | |
| HyLo-Llama-8MLA8M2Teacher=8B, KV cache=7.8%, Training Context Length=8K, Backbone Model=Llama-3.2-1B2026.04 | 59 | 0.3 | 0.1 | 0 | |
| HyLo-Llama-8MLA8M2Teacher=8B, KV cache=7.8%, Training Context Length=64K, Backbone Model=Llama-3.2-1B2026.04 | 59 | 52.5 | 45.5 | 38.8 | |
| HyLo-Qwen-7MLA21M2Teacher=8B, KV cache=3.9%, Training Context Length=8K2026.04 | 58.7 | 41.1 | 27.5 | 14.6 | |
| HyLo-Qwen-7MLA21M2Teacher=8B, KV cache=3.9%, Training Context Length=64K2026.04 | 56.5 | 49 | 38.4 | 27.8 | |
| HyLo-Llama-4MLA12GDNTeacher=8B, KV cache=3.9%, Training Context Length=8K, Backbone Model=Llama-3.2-1B2026.04 | 55.1 | 11.9 | 2.4 | 0.8 | |
| HyLo-Llama-4MLA12M2Teacher=8B, KV cache=3.9%, Training Context Length=64K, Backbone Model=Llama-3.2-1B2026.04 | 53.3 | 46.7 | 40.4 | 37.9 | |
| HyLo-Llama-4MLA12M2Teacher=8B, KV cache=3.9%, Training Context Length=8K, Backbone Model=Llama-3.2-1B2026.04 | 53.1 | 10.6 | 2 | 0.5 | |
| HyLo-Llama-4MLA12GDNTeacher=8B, KV cache=3.9%, Training Context Length=64K, Backbone Model=Llama-3.2-1B2026.04 | 52.5 | 48.3 | 44.5 | 40.8 | |
| Zebra-Llama 3B (6MLA-22M2)Teacher=8B, KV cache=2.0%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 42.5 | 0.4 | 0.5 | 0.3 | |
| Mamba in Llama-3B-50%Teacher=70B, KV cache=50.0%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 37 | 1 | 0 | 0 | |
| Hype Net (7FA21LA)Teacher=1.7B, KV cache=25%2026.04 | 36.4 | 31.3 | 23.8 | 16.4 | |
| Zebra-Llama 3B (14MLA-14M2)Teacher=8B, KV cache=4.7%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 35.1 | 13.3 | 6.3 | 4.2 | |
| MambaInLlama-1B-50%Teacher=8B, KV cache=50%, Backbone Model=Llama-3.2-1B2026.04 | 18.9 | 3 | 1 | 0 | |
| Zebra-Llama-1B (4MLA-12M2)Teacher=8B, KV cache=4%, Backbone Model=Llama-3.2-1B2026.04 | 12.3 | 6.8 | 3.7 | 0.1 | |
| Llamba 3BTeacher=3B+70B, KV cache=0.0%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 3.5 | 0 | 0 | 0 | |
| Llamba 1BTeacher=1B+70B, KV cache=0%, Backbone Model=Llama-3.2-1B2026.04 | 2.9 | 0 | 0 | 0 | |
| Zebra-Llama-1B (8MLA-8M2)Teacher=8B, KV cache=7.8%, Backbone Model=Llama-3.2-1B2026.04 | 0.5 | 0 | 0.1 | 0 |