Long-context understanding on RULER 32K
94.48AccuracyQwen2.5-14B
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen2.5-14BModel=Qwen2.5-14B2025.11 | 94.48 | |
| RetroInferModel=Qwen2.5-14B2025.11 | 94.41 | |
| LiteCache + HATAModel=Qwen2.5-14B2025.11 | 94.22 | |
| HATA AlgoModel=Qwen2.5-14B2025.11 | 94.13 | |
| DenseBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 93.01 | |
| CompactAttention-FPBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 92.41 | |
| Dense AttentionModel=Llama-3.1-8B, Sparsity=Dense, Deployment Phase=Prefill Phase2025.12 | 92.33 | |
| Dense AttentionModel=Llama-3.1-8B, Sparsity=Dense, Deployment Phase=Prefill + Decode Phase2025.12 | 92.33 | |
| XAttentionBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 92.29 | |
| InfiniGenModel=Qwen2.5-14B2025.11 | 92.19 | |
| BLASSTModel=Qwen3-8B, Sparsity=~75%, Deployment Phase=Prefill Phase2025.12 | 92.11 | |
| BLASSTModel=Qwen3-8B, Sparsity=~50%, Deployment Phase=Prefill Phase2025.12 | 92.08 | |
| BLASSTModel=Qwen3-8B, Sparsity=~50%, Deployment Phase=Prefill + Decode Phase2025.12 | 92.07 | |
| Dense AttentionBackbone=Qwen3-8B, Attention Method=Dense, Phase=Decode2025.12 | 91.9 | |
| Dense AttentionModel=Qwen3-8B, Sparsity=Dense, Deployment Phase=Prefill Phase2025.12 | 91.9 | |
| Dense AttentionModel=Qwen3-8B, Sparsity=Dense, Deployment Phase=Prefill + Decode Phase2025.12 | 91.9 | |
| BLASSTModel=Llama-3.1-8B, Sparsity=~50%, Deployment Phase=Prefill Phase2025.12 | 91.81 | |
| BLASSTModel=Llama-3.1-8B, Sparsity=~50%, Deployment Phase=Prefill + Decode Phase2025.12 | 91.79 | |
| BLASSTModel=Qwen3-8B, Sparsity=~75%, Deployment Phase=Prefill + Decode Phase2025.12 | 91.74 | |
| BLASSTModel=Llama-3.1-8B, Sparsity=~75%, Deployment Phase=Prefill Phase2025.12 | 91.67 | |
| BLASSTModel=Llama-3.1-8B, Sparsity=~75%, Deployment Phase=Prefill + Decode Phase2025.12 | 91.67 | |
| BLASSTBackbone=Qwen3-8B, Attention Method=BLASST, Sparsity=~50%, Phase=Decode2025.12 | 91.55 | |
| FlashPrefillBackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 91.55 | |
| SeerAttentionBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 90.23 | |
| XAttentionBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 89.16 | |
| FlashPrefillBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 88.99 | |
| CompactAttention-SABackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 88.92 | |
| CompactAttention-FPBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 88.77 | |
| DenseBackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 88.23 | |
| RocketKVBackbone=Qwen3-8B, Attention Method=RocketKV, Phase=Decode2025.12 | 87.89 | |
| QUOKABackbone=LLaMA-3.1-8B-Instruct, Chunk Size=10242026.05 | 83.52 | |
| QUOKABackbone=Qwen3-30B-A3B-Instruct-2507, Chunk Size=10242026.05 | 81.96 | |
| HATA AlgoModel=Llama3-8B2025.11 | 80.94 | |
| Llama3-8BModel=Llama3-8B2025.11 | 80.8 | |
| LiteCache + HATAModel=Llama3-8B2025.11 | 80.7 | |
| RetroInferModel=Llama3-8B2025.11 | 80.64 | |
| InfiniGenModel=Llama3-8B2025.11 | 76.76 | |
| QuestBackbone=Qwen3-8B, Attention Method=Quest, Phase=Decode2025.12 | 56.23 |