List your company
ResearchFormerly WizWand

Downstream Performance Evaluation on HELMET (Long-Context)

33.1ScoreThriftAttention
9.1815.3921.627.81May 21, 2026
Updated 5mo ago

Evaluation results

MethodLinks
ThriftAttentionModel=Ministral 3 (8B), Budget (k)=10%2026.0533.1100.6
FP16Model=Ministral 3 (8B)2026.0533100
ThriftAttentionModel=Ministral 3 (8B), Budget (k)=25%2026.0533100
ThriftAttentionModel=Ministral 3 (8B), Budget (k)=5%2026.0532.999.4
FP16Model=Llama3.1 (8B)2026.0531.2100
ThriftAttentionModel=Llama3.1 (8B), Budget (k)=25%2026.0529.188.3
FP16Model=Llama3.2 (3B)2026.0528.9100
ThriftAttentionModel=Llama3.1 (8B), Budget (k)=5%2026.0528.786.1
ThriftAttentionModel=Llama3.1 (8B), Budget (k)=10%2026.0528.685.6
ThriftAttentionModel=Qwen3 (8B), Budget (k)=10%2026.0526.8106.8
ThriftAttentionModel=Llama3.2 (3B), Budget (k)=25%2026.0526.687.6
FP16Model=Qwen3 (8B)2026.0525.9100
ThriftAttentionModel=Llama3.2 (3B), Budget (k)=10%2026.0525.179.6
FP16Model=Qwen3 (4B)2026.0524.9100
ThriftAttentionModel=Llama3.2 (3B), Budget (k)=5%2026.0524.777.6
ThriftAttentionModel=Qwen3 (4B), Budget (k)=25%2026.0524.697.5
ThriftAttentionModel=Qwen3 (8B), Budget (k)=5%2026.0524.690.2
ThriftAttentionModel=Qwen3 (8B), Budget (k)=25%2026.0524.690.2
ThriftAttentionModel=Qwen3 (4B), Budget (k)=10%2026.0523.890.8
ThriftAttentionModel=Qwen3 (4B), Budget (k)=5%2026.0523.285.7
FP4Model=Ministral 3 (8B)2026.0515.90
FP4Model=Llama3.1 (8B)2026.0513.20
FP4Model=Qwen3 (4B)2026.05130
FP4Model=Qwen3 (8B)2026.0512.70
FP4Model=Llama3.2 (3B)2026.0510.10