List your company
ResearchFormerly WizWand

Time To First Token (TTFT) on LLM Prefix-based Inference

51TTFT (ms)FlashRT
-29.24512.381,0541,595.62Jun 18, 2026
Updated 4mo ago

Evaluation results

MethodLinks
FlashRTprefix_length=2048, caching_mode=capsule (with reuse)2026.0651
FlashRTprefix_length=4096, caching_mode=capsule (with reuse)2026.0653
FlashRTprefix_length=8192, caching_mode=capsule (with reuse)2026.0654
vLLMprefix_length=4096, caching_mode=APC (with reuse)2026.0676
vLLMprefix_length=8192, caching_mode=APC (with reuse)2026.06120
vLLMprefix_length=2048, caching_mode=APC (with reuse)2026.06143
FlashRTprefix_length=2048, caching_mode=no reuse (cold)2026.06200
FlashRTprefix_length=4096, caching_mode=no reuse (cold)2026.06365
vLLMprefix_length=2048, caching_mode=no reuse (cold)2026.06519
FlashRTprefix_length=8192, caching_mode=no reuse (cold)2026.06723
vLLMprefix_length=4096, caching_mode=no reuse (cold)2026.061,026
vLLMprefix_length=8192, caching_mode=no reuse (cold)2026.062,057