Time To First Token (TTFT) Latency on Llama 8B 4K Shared Context 3.1
24.7TTFT (ms)Q4 cache reload
Evaluation results
| Method | Links | |
|---|---|---|
| Q4 cache reloadPrecision=Q4, Handoff Mechanism=cache reload2026.05 | 24.7 | |
| FP16 cache copyPrecision=FP16, Handoff Mechanism=cache copy2026.05 | 26.1 | |
| QKVShare uniform Q4 handoffPrecision=Q4, Handoff Mechanism=handoff2026.05 | 152.5 | |
| FP16 re-prefillPrecision=FP16, Handoff Mechanism=re-prefill2026.05 | 565.3 |