Long-context downstream performance on RULER
86.6ScoreThriftAttention
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ThriftAttentionModel=Ministral 3 (8B), Budget (k)=5%2026.05 | 86.6 | 106 | |
| ThriftAttentionModel=Ministral 3 (8B), Budget (k)=25%2026.05 | 86.5 | 105.7 | |
| ThriftAttentionModel=Ministral 3 (8B), Budget (k)=10%2026.05 | 86.4 | 105.3 | |
| FP16Model=Llama3.1 (8B)2026.05 | 85.9 | 100 | |
| FP16Model=Ministral 3 (8B)2026.05 | 84.8 | 100 | |
| FP16Model=Qwen3 (8B)2026.05 | 84 | 100 | |
| ThriftAttentionModel=Qwen3 (8B), Budget (k)=25%2026.05 | 81.9 | 94.1 | |
| ThriftAttentionModel=Qwen3 (8B), Budget (k)=10%2026.05 | 81.5 | 93 | |
| ThriftAttentionModel=Qwen3 (8B), Budget (k)=5%2026.05 | 80.6 | 90.4 | |
| ThriftAttentionModel=Llama3.1 (8B), Budget (k)=25%2026.05 | 80 | 87.6 | |
| FP16Model=Qwen3 (4B)2026.05 | 79.5 | 100 | |
| ThriftAttentionModel=Llama3.1 (8B), Budget (k)=10%2026.05 | 77.8 | 82.9 | |
| ThriftAttentionModel=Llama3.1 (8B), Budget (k)=5%2026.05 | 77 | 81.2 | |
| ThriftAttentionModel=Qwen3 (4B), Budget (k)=25%2026.05 | 76.8 | 92.3 | |
| ThriftAttentionModel=Qwen3 (4B), Budget (k)=10%2026.05 | 75.6 | 88.9 | |
| ThriftAttentionModel=Qwen3 (4B), Budget (k)=5%2026.05 | 74.9 | 86.9 | |
| FP16Model=Llama3.2 (3B)2026.05 | 72.8 | 100 | |
| ThriftAttentionModel=Llama3.2 (3B), Budget (k)=25%2026.05 | 57.7 | 62.9 | |
| ThriftAttentionModel=Llama3.2 (3B), Budget (k)=10%2026.05 | 54.8 | 55.8 | |
| FP4Model=Ministral 3 (8B)2026.05 | 54.8 | 0 | |
| ThriftAttentionModel=Llama3.2 (3B), Budget (k)=5%2026.05 | 54.3 | 54.5 | |
| FP4Model=Qwen3 (8B)2026.05 | 48.4 | 0 | |
| FP4Model=Qwen3 (4B)2026.05 | 44.5 | 0 | |
| FP4Model=Llama3.1 (8B)2026.05 | 38.5 | 0 | |
| FP4Model=Llama3.2 (3B)2026.05 | 32.1 | 0 |