Common Sense Reasoning on ARC, ARC-Easy, HellaSwag, OpenBookQA, PIQA, RACE, WinoGrande
47.3ARC AccuracyHyLo-Llama-14MLA14GDN
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| HyLo-Llama-14MLA14GDNTeacher=8B, KV cache=4.7%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 47.3 | 72.6 | 69.5 | 40 | 76.3 | 41.5 | 67.3 | 59.2 | |
| Mamba in Llama-3B-50%Teacher=70B, KV cache=50.0%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 47.1 | 74 | 69 | 38.4 | 75.9 | 40.1 | 66.5 | 58.7 | |
| HyLo-Llama-14MLA14M2Teacher=8B, KV cache=4.7%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 46.3 | 73 | 68.7 | 40.4 | 75.9 | 40.4 | 67.7 | 58.9 | |
| Llamba 3BTeacher=3B+70B, KV cache=0.0%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 45.7 | 73.8 | 73.3 | 42.4 | 78 | 40.1 | 70 | 60.5 | |
| Zebra-Llama 3B (14MLA-14M2)Teacher=8B, KV cache=4.7%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 45.7 | 71.8 | 68.6 | 38.6 | 75.7 | 40.9 | 64.4 | 58 | |
| M1KV cache=21.4%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 45.6 | 72.6 | 61.5 | 39.4 | 73.3 | 35.9 | 64.9 | 56.2 | |
| HyLo-Llama-6MLA22M2Teacher=8B, KV cache=2.0%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 45.6 | 72.4 | 67.8 | 38.4 | 76.1 | 39.7 | 66.8 | 58.1 | |
| HyLo-Llama-6MLA22GDNTeacher=8B, KV cache=2.0%, Training Context Length=8K, Backbone=Llama-3.2-3B2026.04 | 45.4 | 71.9 | 69.3 | 42.4 | 76.6 | 39.7 | 67.6 | 59 | |
| HyLo-Llama-14MLA14GDNTeacher=8B, KV cache=4.7%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 45.1 | 72 | 68.2 | 39.4 | 76.1 | 40.9 | 63.8 | 57.9 | |
| Zebra-Llama 3B (6MLA-22M2)Teacher=8B, KV cache=2.0%, Training Context Length=Baseline, Backbone=Llama-3.2-3B2026.04 | 44.7 | 70.8 | 67.7 | 38.8 | 75.6 | 39.4 | 64.5 | 57.4 | |
| HyLo-Llama-14MLA14M2Teacher=8B, KV cache=4.7%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 44.1 | 71.2 | 67.3 | 39.6 | 75.4 | 40 | 64.3 | 57.4 | |
| HyLo-Llama-6MLA22GDNTeacher=8B, KV cache=2.0%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 43.7 | 69.5 | 67.9 | 38.6 | 75.9 | 39.8 | 64.9 | 57.2 | |
| HyLo-Llama-6MLA22M2Teacher=8B, KV cache=2.0%, Training Context Length=64K, Backbone=Llama-3.2-3B2026.04 | 43.5 | 69.7 | 66.2 | 38.8 | 75.5 | 38.6 | 64.3 | 56.7 |