Time To First Token (TTFT) on LLM Prefix-based Inference
51TTFT (ms)FlashRT
Evaluation Results
| Method | Links | |
|---|---|---|
| FlashRTprefix_length=2048, caching_mode=capsule (with reuse)2026.06 | 51 | |
| FlashRTprefix_length=4096, caching_mode=capsule (with reuse)2026.06 | 53 | |
| FlashRTprefix_length=8192, caching_mode=capsule (with reuse)2026.06 | 54 | |
| vLLMprefix_length=4096, caching_mode=APC (with reuse)2026.06 | 76 | |
| vLLMprefix_length=8192, caching_mode=APC (with reuse)2026.06 | 120 | |
| vLLMprefix_length=2048, caching_mode=APC (with reuse)2026.06 | 143 | |
| FlashRTprefix_length=2048, caching_mode=no reuse (cold)2026.06 | 200 | |
| FlashRTprefix_length=4096, caching_mode=no reuse (cold)2026.06 | 365 | |
| vLLMprefix_length=2048, caching_mode=no reuse (cold)2026.06 | 519 | |
| FlashRTprefix_length=8192, caching_mode=no reuse (cold)2026.06 | 723 | |
| vLLMprefix_length=4096, caching_mode=no reuse (cold)2026.06 | 1,026 | |
| vLLMprefix_length=8192, caching_mode=no reuse (cold)2026.06 | 2,057 |