List your company
ResearchFormerly WizWand

Language Modeling on OWT Pile

5.23Decode LatencyLongformer + SFA (k=8)
4.7567.955511.15514.3545Mar 17, 2026
Updated 7mo ago

Evaluation results

MethodLinks
Longformer + SFA (k=8)Backbone=GPT-2 124M, Method Category=Token Sparse (Training), Latency Context Length=128k, Evaluation Protocol=Training, SFA k parameter=82026.035.236.1819.3
LongformerBackbone=GPT-2 124M, Method Category=Token Sparse (Training), Latency Context Length=128k, Evaluation Protocol=Training2026.036.757.9318.73
SnapKV + SFA (k=8)Backbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free, SFA k parameter=82026.036.929.4119.44
RoutingBackbone=GPT-2 124M, Method Category=Token Sparse (Training), Latency Context Length=128k, Evaluation Protocol=Training2026.037.928.3718.64
Loki + SFA (k=8)Backbone=GPT-2 124M, Method Category=Low-rank keys (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free, SFA k parameter=82026.039.099.4119.29
PerformerBackbone=GPT-2 124M, Method Category=Kernel Method, Latency Context Length=128k2026.039.437.9319.72
SnapKVBackbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.039.8816.8617.91
QuestBackbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.0310.8416.8617.95
LokiBackbone=GPT-2 124M, Method Category=Low-rank keys (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.0311.3916.8617.82
H2OBackbone=GPT-2 124M, Method Category=KV-pruning (Training-free), Latency Context Length=128k, Evaluation Protocol=Training-free2026.0313.3216.8618.02
SFABackbone=GPT-2 124M, Method Category=Feature Sparse Attention, Latency Context Length=128k2026.0314.129.4118.17
Dense (full)Backbone=GPT-2 124M, Method Category=Full Attention Baseline, Latency Context Length=128k2026.0317.0816.8617.29