List your company
ResearchFormerly WizWand

Time to First Token on Llama 3.1 8B Q4 weights

204TTFT (ms)vllm-mlx
-1,69311,111.7523,916.536,721.25Feb 17, 2026
Updated 8mo ago

Evaluation results

MethodLinks
vllm-mlxContext Length=2K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02204
vllm-mlxContext Length=8K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02244
agent-memoryContext Length=1K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02260
agent-memoryContext Length=1K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02260
vllm-mlxContext Length=1K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02273
agent-memoryContext Length=2K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02274
agent-memoryContext Length=2K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02274
agent-memoryContext Length=4K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02290
agent-memoryContext Length=4K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02290
vllm-mlxContext Length=4K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02290
agent-memoryContext Length=8K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02331
agent-memoryContext Length=8K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02331
vllm-mlxContext Length=16K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02371
agent-memoryContext Length=16K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02431
agent-memoryContext Length=16K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02431
vllm-mlxContext Length=2K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.022,204
vllm-mlxContext Length=1K, Cache State=Restart, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.022,262
vllm-mlxContext Length=1K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.022,289
agent-memoryContext Length=1K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.022,500
vllm-mlxContext Length=2K, Cache State=Restart, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.024,134
vllm-mlxContext Length=4K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.024,394
agent-memoryContext Length=2K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.024,530
vllm-mlxContext Length=4K, Cache State=Restart, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.028,079
vllm-mlxContext Length=8K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.029,472
agent-memoryContext Length=4K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.0210,235
agent-memoryContext Length=8K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.0221,536
vllm-mlxContext Length=16K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.0237,935
agent-memoryContext Length=16K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.0247,629