Long-context performance evaluation on RULER
96AccuracyFP16
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FP16Backbone=LLaMA-3-8B-Instruct, BPA=162026.07 | 96 | — | — | — | — | — | — | |
| GSRQBackbone=LLaMA-3-8B-Instruct, BPA=22026.07 | 96 | — | — | — | — | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Context Length=4K2026.02 | 95.31 | — | — | — | — | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Context Length=8K2026.02 | 95.31 | — | — | — | — | — | — | |
| Sketch&WalkModel=Llama-3.1-8B-Instruct, Context Length=8K2026.02 | 95 | — | — | — | — | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Context Length=16K2026.02 | 94.06 | — | — | — | — | — | — | |
| KIVIBackbone=LLaMA-3-8B-Instruct, BPA=22026.07 | 94 | — | — | — | — | — | — | |
| VQLLMBackbone=LLaMA-3-8B-Instruct, BPA=22026.07 | 94 | — | — | — | — | — | — | |
| Sketch&WalkModel=Llama-3.1-8B-Instruct, Context Length=4K2026.02 | 93.64 | — | — | — | — | — | — | |
| Sketch&WalkModel=Llama-3.1-8B-Instruct, Context Length=16K2026.02 | 93.13 | — | — | — | — | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Context Length=32K2026.02 | 89.69 | — | — | — | — | — | — | |
| Sketch&WalkModel=Llama-3.1-8B-Instruct, Context Length=32K2026.02 | 87.5 | — | — | — | — | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Context Length=64K2026.02 | 82.81 | — | — | — | — | — | — | |
| Sketch&WalkModel=Llama-3.1-8B-Instruct, Context Length=64K2026.02 | 81.5 | — | — | — | — | — | — | |
| GSRQBackbone=LLaMA-3-8B-Instruct, BPA=12026.07 | 72 | — | — | — | — | — | — | |
| VQLLMBackbone=LLaMA-3-8B-Instruct, BPA=12026.07 | 20 | — | — | — | — | — | — | |
| KIVIBackbone=LLaMA-3-8B-Instruct, BPA=12026.07 | 13 | — | — | — | — | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Context Length=Average2026.02 | — | 91.44 | — | — | — | — | — | |
| FastKVPrefill=60%, KV=10%, Backbone=LLaMA-3.1-8B-Instruct2025.02 | — | 75.6 | 77.8 | 77.3 | 77.2 | 77.4 | 68.2 | |
| Full-contextPrefill=100%, KV=100%, Backbone=LLaMA-3.1-8B-Instruct2025.02 | — | 86 | 90.1 | 95 | 83.4 | 85.5 | 76.3 | |
| GemFilterPrefill=51%, KV=10%, Backbone=LLaMA-3.1-8B-Instruct2025.02 | — | 69.6 | 69.7 | 68.2 | 70.4 | 69.8 | 69.8 | |
| H2OPrefill=100%, KV=10%, Backbone=LLaMA-3.1-8B-Instruct2025.02 | — | — | 27.1 | — | — | — | — | |
| PyramidInferPrefill=60%, KV=60%, Backbone=LLaMA-3.1-8B-Instruct2025.02 | — | — | 66.5 | — | — | — | — | |
| Sketch&WalkModel=Llama-3.1-8B-Instruct, Context Length=Average2026.02 | — | 90.15 | — | — | — | — | — | |
| SnapKVPrefill=100%, KV=10%, Backbone=LLaMA-3.1-8B-Instruct2025.02 | — | 73.6 | 75.6 | 76.8 | 72.9 | 75 | 67.7 | |
| StreamingLLMPrefill=100%, KV=10%, Backbone=LLaMA-3.1-8B-Instruct2025.02 | — | 18.6 | 15 | 21.5 | 24.4 | 16.9 | 15 |