Zero-shot Classification on Evaluation Suite (PiQA, LAMBDA, ARC-e, ARC-c, HellaS)
5.23Decode LatencyLongformer + SFA (k=8)
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Longformer + SFA (k=8)Backbone=GPT-2 124M, Method Category=Token Sparse (Training), Latency Context Length=128k, Evaluation Protocol=Training, SFA k parameter=82026.03 | 5.23 | 6.18 | 40.75 | 20.54 | 26.39 | 6.63 | 17.24 | 22.31 | |
| LongformerBackbone=GPT-2 124M, Method Category=Token Sparse (Training), Latency Context Length=128k, Evaluation Protocol=Training2026.03 | 6.75 | 7.93 | 41.28 | 21.27 | 28.02 | 7.01 | 18.92 | 23.3 | |
| SnapKV + SFA (k=8)Backbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free, SFA k parameter=82026.03 | 6.92 | 9.41 | 39.99 | 20.24 | 27.13 | 6.83 | 17.74 | 22.39 | |
| RoutingBackbone=GPT-2 124M, Method Category=Token Sparse (Training), Latency Context Length=128k, Evaluation Protocol=Training2026.03 | 7.92 | 8.37 | 41.39 | 21.08 | 28.31 | 7.11 | 18.89 | 23.35 | |
| Loki + SFA (k=8)Backbone=GPT-2 124M, Method Category=Low-rank keys (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free, SFA k parameter=82026.03 | 9.09 | 9.41 | 40.83 | 20.04 | 27.85 | 7.13 | 18.03 | 22.78 | |
| PerformerBackbone=GPT-2 124M, Method Category=Kernel Method, Latency Context Length=128k2026.03 | 9.43 | 7.93 | 39.83 | 19.11 | 26.72 | 6.77 | 15.38 | 21.56 | |
| SnapKVBackbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.03 | 9.88 | 16.86 | 42.49 | 21.92 | 28.43 | 8.01 | 19.38 | 24.05 | |
| QuestBackbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.03 | 10.84 | 16.86 | 42.34 | 20.79 | 28.3 | 7.82 | 18.83 | 23.62 | |
| LokiBackbone=GPT-2 124M, Method Category=Low-rank keys (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.03 | 11.39 | 16.86 | 42.1 | 21.29 | 28.01 | 7.99 | 19.24 | 23.73 | |
| H2OBackbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.03 | 13.32 | 16.86 | 41.81 | 20.55 | 27.04 | 7.38 | 18.75 | 23.11 | |
| SFABackbone=GPT-2 124M, Method Category=Feature Sparse Attention, Latency Context Length=128k2026.03 | 14.12 | 9.41 | 41.62 | 21.03 | 28.41 | 7.39 | 19.26 | 23.54 | |
| Dense (full)Backbone=GPT-2 124M, Method Category=Full Attention Baseline, Latency Context Length=128k2026.03 | 17.08 | 16.86 | 42.74 | 22.78 | 28.35 | 8.12 | 19.61 | 24.32 |