Needle-in-a-Haystack 4x Original Context
100AccuracyTransactional Attention
Evaluation Results
| Method | Links | |
|---|---|---|
| Transactional AttentionRetention Budget (K)=16, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| Transactional AttentionRetention Budget (K)=32, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| Transactional AttentionRetention Budget (K)=64, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| Transactional AttentionRetention Budget (K)=128, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| Transactional AttentionRetention Budget (K)=256, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| TOVARetention Budget (K)=128, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| TOVARetention Budget (K)=256, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| SnapKVRetention Budget (K)=128, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| SnapKVRetention Budget (K)=256, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 100 | |
| SmolLM-DroPERoPE scaling method=DroPE2025.12 | 55 | |
| PyramidKVRetention Budget (K)=256, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 46 | |
| SmolLM + LongRoPE2RoPE scaling method=LongRoPE22025.12 | 26.2 | |
| SmolLM + YaRNRoPE scaling method=YaRN2025.12 | 25.62 | |
| SmolLM + RoPE-NTKRoPE scaling method=RoPE-NTK2025.12 | 14.37 | |
| TOVARetention Budget (K)=16, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| TOVARetention Budget (K)=32, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| TOVARetention Budget (K)=64, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| H2ORetention Budget (K)=16, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| H2ORetention Budget (K)=32, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| H2ORetention Budget (K)=64, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| H2ORetention Budget (K)=128, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| H2ORetention Budget (K)=256, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| SnapKVRetention Budget (K)=16, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| SnapKVRetention Budget (K)=32, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| SnapKVRetention Budget (K)=64, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| StreamingLLMRetention Budget (K)=16, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| StreamingLLMRetention Budget (K)=32, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| StreamingLLMRetention Budget (K)=64, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| StreamingLLMRetention Budget (K)=128, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| StreamingLLMRetention Budget (K)=256, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| PyramidKVRetention Budget (K)=16, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| PyramidKVRetention Budget (K)=32, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| PyramidKVRetention Budget (K)=64, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| PyramidKVRetention Budget (K)=128, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 | |
| DynamicKVRetention Budget (K)=16, Base Model=Llama-3.2-1B, Context Length=4K2026.04 | 0 |