Downstream Performance Evaluation on HELMET (Long-Context)
33.1ScoreThriftAttention
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ThriftAttentionModel=Ministral 3 (8B), Budget (k)=10%2026.05 | 33.1 | 100.6 | |
| FP16Model=Ministral 3 (8B)2026.05 | 33 | 100 | |
| ThriftAttentionModel=Ministral 3 (8B), Budget (k)=25%2026.05 | 33 | 100 | |
| ThriftAttentionModel=Ministral 3 (8B), Budget (k)=5%2026.05 | 32.9 | 99.4 | |
| FP16Model=Llama3.1 (8B)2026.05 | 31.2 | 100 | |
| ThriftAttentionModel=Llama3.1 (8B), Budget (k)=25%2026.05 | 29.1 | 88.3 | |
| FP16Model=Llama3.2 (3B)2026.05 | 28.9 | 100 | |
| ThriftAttentionModel=Llama3.1 (8B), Budget (k)=5%2026.05 | 28.7 | 86.1 | |
| ThriftAttentionModel=Llama3.1 (8B), Budget (k)=10%2026.05 | 28.6 | 85.6 | |
| ThriftAttentionModel=Qwen3 (8B), Budget (k)=10%2026.05 | 26.8 | 106.8 | |
| ThriftAttentionModel=Llama3.2 (3B), Budget (k)=25%2026.05 | 26.6 | 87.6 | |
| FP16Model=Qwen3 (8B)2026.05 | 25.9 | 100 | |
| ThriftAttentionModel=Llama3.2 (3B), Budget (k)=10%2026.05 | 25.1 | 79.6 | |
| FP16Model=Qwen3 (4B)2026.05 | 24.9 | 100 | |
| ThriftAttentionModel=Llama3.2 (3B), Budget (k)=5%2026.05 | 24.7 | 77.6 | |
| ThriftAttentionModel=Qwen3 (4B), Budget (k)=25%2026.05 | 24.6 | 97.5 | |
| ThriftAttentionModel=Qwen3 (8B), Budget (k)=5%2026.05 | 24.6 | 90.2 | |
| ThriftAttentionModel=Qwen3 (8B), Budget (k)=25%2026.05 | 24.6 | 90.2 | |
| ThriftAttentionModel=Qwen3 (4B), Budget (k)=10%2026.05 | 23.8 | 90.8 | |
| ThriftAttentionModel=Qwen3 (4B), Budget (k)=5%2026.05 | 23.2 | 85.7 | |
| FP4Model=Ministral 3 (8B)2026.05 | 15.9 | 0 | |
| FP4Model=Llama3.1 (8B)2026.05 | 13.2 | 0 | |
| FP4Model=Qwen3 (4B)2026.05 | 13 | 0 | |
| FP4Model=Qwen3 (8B)2026.05 | 12.7 | 0 | |
| FP4Model=Llama3.2 (3B)2026.05 | 10.1 | 0 |