Time to First Token on Llama 3.1 8B Q4 weights
204TTFT (ms)vllm-mlx
Evaluation Results
| Method | Links | |
|---|---|---|
| vllm-mlxContext Length=2K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 204 | |
| vllm-mlxContext Length=8K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 244 | |
| agent-memoryContext Length=1K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 260 | |
| agent-memoryContext Length=1K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 260 | |
| vllm-mlxContext Length=1K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 273 | |
| agent-memoryContext Length=2K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 274 | |
| agent-memoryContext Length=2K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 274 | |
| agent-memoryContext Length=4K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 290 | |
| agent-memoryContext Length=4K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 290 | |
| vllm-mlxContext Length=4K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 290 | |
| agent-memoryContext Length=8K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 331 | |
| agent-memoryContext Length=8K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 331 | |
| vllm-mlxContext Length=16K, Cache State=Prefix, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 371 | |
| agent-memoryContext Length=16K, Cache State=Warm, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 431 | |
| agent-memoryContext Length=16K, Cache State=Restart, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 431 | |
| vllm-mlxContext Length=2K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 2,204 | |
| vllm-mlxContext Length=1K, Cache State=Restart, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 2,262 | |
| vllm-mlxContext Length=1K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 2,289 | |
| agent-memoryContext Length=1K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 2,500 | |
| vllm-mlxContext Length=2K, Cache State=Restart, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 4,134 | |
| vllm-mlxContext Length=4K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 4,394 | |
| agent-memoryContext Length=2K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 4,530 | |
| vllm-mlxContext Length=4K, Cache State=Restart, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 8,079 | |
| vllm-mlxContext Length=8K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 9,472 | |
| agent-memoryContext Length=4K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 10,235 | |
| agent-memoryContext Length=8K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 21,536 | |
| vllm-mlxContext Length=16K, Cache State=Cold, KV Cache Precision=FP16, Persistence=volatile, Batch Size=1, Streaming=true2026.02 | 37,935 | |
| agent-memoryContext Length=16K, Cache State=Cold, KV Cache Precision=Q4, Persistence=disk-persistent, Batch Size=1, Streaming=true2026.02 | 47,629 |