Time To First Token (TTFT) Latency on Llama 8B 8K Shared Context 3.1
24.8TTFT (ms)Q4 cache reload
Evaluation results
| Method | Links | |
|---|---|---|
| Q4 cache reloadPrecision=Q4, Handoff Mechanism=cache reload2026.05 | 24.8 | |
| FP16 cache copyPrecision=FP16, Handoff Mechanism=cache copy2026.05 | 26.5 | |
| QKVShare uniform Q4 handoffPrecision=Q4, Handoff Mechanism=handoff2026.05 | 397.1 | |
| FP16 re-prefillPrecision=FP16, Handoff Mechanism=re-prefill2026.05 | 1,029.7 |