Language Modeling on OpenWebText2
5.45Perplexity (PPL)Full KV
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Full KVcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=Full2026.05 | 5.45 | — | |
| Meta-Softcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.68 | — | |
| Judge Qcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.71 | — | |
| Full KVcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=Full2026.05 | 5.72 | — | |
| ZeroMergecontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.75 | — | |
| H2Ocontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.82 | — | |
| SnapKVcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.83 | — | |
| Meta-Softcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.87 | — | |
| Judge Qcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.94 | — | |
| ZeroMergecontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 5.98 | — | |
| SnapKVcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 6.09 | — | |
| H2Ocontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 6.12 | — | |
| AIR-MoESize=Large2026.05 | 16.65 | 11.2 | |
| Coarse MoESize=Large2026.05 | 16.98 | 11.3 | |
| PEERSize=Large2026.05 | 17.88 | 11.2 | |
| Hierarchical MoESize=Large2026.05 | 18.17 | 11.3 | |
| AIR-MoESize=Medium2026.05 | 20.51 | 3.6 | |
| Coarse MoESize=Medium2026.05 | 21.25 | 3.5 | |
| PEERSize=Medium2026.05 | 21.3 | 3.5 | |
| Hierarchical MoESize=Medium2026.05 | 23.07 | 3.5 | |
| Standard Granular MoESize=Small2026.05 | 31.56 | 648.3 | |
| AIR-MoESize=Small2026.05 | 32.14 | 505 | |
| PEERSize=Small2026.05 | 33.1 | 505.6 | |
| Hierarchical MoESize=Small2026.05 | 36.05 | 502.6 | |
| Coarse MoESize=Small2026.05 | 36.05 | 493.9 |