LLM Inference Efficiency on Uniform Sampled Documents
191.7TTFT (ms)Lazy-Attn
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Lazy-AttnHardware=H100 (High BW)2026.06 | 191.7 | 6.2 | |
| CacheBlendHardware=H100 (High BW)2026.06 | 274.8 | 4.3 | |
| Lazy-AttnHardware=A100 (Mid BW)2026.06 | 372.5 | 6.9 | |
| CacheBlendHardware=A100 (Mid BW)2026.06 | 565.3 | 4.5 | |
| Lazy-AttnHardware=A40 (Low BW)2026.06 | 675.2 | 6.1 | |
| CacheBlendHardware=A40 (Low BW)2026.06 | 1,150.6 | 3.6 | |
| Prefix CachingHardware=H100 (High BW)2026.06 | 1,196.8 | 1 | |
| Prefix CachingHardware=A100 (Mid BW)2026.06 | 2,580.4 | 1 | |
| Prefix CachingHardware=A40 (Low BW)2026.06 | 4,150.5 | 1 |