Time To First Token (TTFT) Latency on Llama 8B 1K Shared Context 3.1
21.7TTFT (ms)FP16 cache copy
Evaluation results
| Method | Links | |
|---|---|---|
| FP16 cache copyPrecision=FP16, Handoff Mechanism=cache copy2026.05 | 21.7 | |
| Q4 cache reloadPrecision=Q4, Handoff Mechanism=cache reload2026.05 | 26.7 | |
| QKVShare uniform Q4 handoffPrecision=Q4, Handoff Mechanism=handoff2026.05 | 130.7 | |
| FP16 re-prefillPrecision=FP16, Handoff Mechanism=re-prefill2026.05 | 150.2 |