# Time to First Token on Llama 3.1 8B Q4 weights Source: [Time to First Token on Llama 3.1 8B Q4 weights]() Collection index: [llms.txt]() Data updated: 2026-03-06T07:28:25.661Z Evaluation of the latency performance for the Llama 3.1 8B model quantized to Q4 weights, specifically measuring the Time to First Token (TTFT). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Time to First Token | | Dataset (including any stated split) | Llama 3.1 8B Q4 weights | | Metric | TTFT (ms) | | Unit | TIME | | Direction | Lower is better | | Results loaded | 28 of 28 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [TTFT (ms)](): Lower is better; unit: TIME. ## Recorded results | Method | TTFT (ms) | Result date | Report paper | | --- | --- | --- | --- | | vllm-mlx | 204 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 244 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 260 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 260 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 273 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 274 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 274 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 290 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 290 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 290 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 331 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 331 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 371 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 431 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 431 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 2204 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 2262 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 2289 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 2500 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 4134 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 4394 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 4530 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 8079 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 9472 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 10235 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 21536 | Not recorded | 69aa5f7c2d603e6de3e4185b | | vllm-mlx | 37935 | Not recorded | 69aa5f7c2d603e6de3e4185b | | agent-memory | 47629 | Not recorded | 69aa5f7c2d603e6de3e4185b | ### vllm-mlx TTFT (ms): 204. Result ID: 69aa81e86dbb7075b953e3e4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"2K","Cache State":"Prefix","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 244. Result ID: 69aa81e86dbb7075b953e3ea. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"8K","Cache State":"Prefix","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 260. Result ID: 69aa81e86dbb7075b953e3d2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"1K","Cache State":"Warm","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 260. Result ID: 69aa81e86dbb7075b953e3d3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"1K","Cache State":"Restart","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 273. Result ID: 69aa81e86dbb7075b953e3e1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"1K","Cache State":"Prefix","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 274. Result ID: 69aa81e86dbb7075b953e3d5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"2K","Cache State":"Warm","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 274. Result ID: 69aa81e86dbb7075b953e3d6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"2K","Cache State":"Restart","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 290. Result ID: 69aa81e86dbb7075b953e3d8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"4K","Cache State":"Warm","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 290. Result ID: 69aa81e86dbb7075b953e3d9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"4K","Cache State":"Restart","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 290. Result ID: 69aa81e86dbb7075b953e3e7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"4K","Cache State":"Prefix","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 331. Result ID: 69aa81e86dbb7075b953e3db. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"8K","Cache State":"Warm","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 331. Result ID: 69aa81e86dbb7075b953e3dc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"8K","Cache State":"Restart","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 371. Result ID: 69aa81e86dbb7075b953e3ec. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"16K","Cache State":"Prefix","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 431. Result ID: 69aa81e86dbb7075b953e3de. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"16K","Cache State":"Warm","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 431. Result ID: 69aa81e86dbb7075b953e3df. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"16K","Cache State":"Restart","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 2204. Result ID: 69aa81e86dbb7075b953e3e3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"2K","Cache State":"Cold","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 2262. Result ID: 69aa81e86dbb7075b953e3e2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"1K","Cache State":"Restart","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 2289. Result ID: 69aa81e86dbb7075b953e3e0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"1K","Cache State":"Cold","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 2500. Result ID: 69aa81e86dbb7075b953e3d1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"1K","Cache State":"Cold","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 4134. Result ID: 69aa81e86dbb7075b953e3e5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"2K","Cache State":"Restart","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 4394. Result ID: 69aa81e86dbb7075b953e3e6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"4K","Cache State":"Cold","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 4530. Result ID: 69aa81e86dbb7075b953e3d4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"2K","Cache State":"Cold","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 8079. Result ID: 69aa81e86dbb7075b953e3e8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"4K","Cache State":"Restart","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 9472. Result ID: 69aa81e86dbb7075b953e3e9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"8K","Cache State":"Cold","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 10235. Result ID: 69aa81e86dbb7075b953e3d7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"4K","Cache State":"Cold","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 21536. Result ID: 69aa81e86dbb7075b953e3da. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"8K","Cache State":"Cold","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### vllm-mlx TTFT (ms): 37935. Result ID: 69aa81e86dbb7075b953e3eb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"16K","Cache State":"Cold","KV Cache Precision":"FP16","Persistence":"volatile","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. ### agent-memory TTFT (ms): 47629. Result ID: 69aa81e86dbb7075b953e3dd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Context Length":"16K","Cache State":"Cold","KV Cache Precision":"Q4","Persistence":"disk-persistent","Batch Size":"1","Streaming":"true"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.