# P99 Per-Token Latency on LLaMA-2 7B (inference) Source: [P99 Per-Token Latency on LLaMA-2 7B (inference)]() Collection index: [llms.txt]() Data updated: 2026-04-28T08:30:44.278Z Evaluation focusing specifically on the P99 per-token latency metric for the LLaMA-2 7B model during inference operations. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Large Language Model Inference | | Dataset (including any stated split) | LLaMA-2 7B (inference) | | Metric | P99 Per-Token Latency (ms) | | Unit | TIME | | Direction | Lower is better | | Results loaded | 33 of 33 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [P99 Per-Token Latency (ms)](): Lower is better; unit: TIME. ## Recorded results | Method | P99 Per-Token Latency (ms) | Result date | Report paper | | --- | --- | --- | --- | | JIT+CUDA | 8.23 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 9.44 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 11.36 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 12.54 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 14.11 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 15.65 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 16.53 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 16.54 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 16.54 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 16.56 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 16.61 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 16.65 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 16.73 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 17.31 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.3 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.38 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.39 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.39 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.41 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.42 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.44 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.46 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.49 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.49 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 18.9 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace | 18.94 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 20.5 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 22.09 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 23.68 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 24.03 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 31.59 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 52.28 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT–LLM | 68.84 | Not recorded | 69f03f4455a3c32af89fa244 | ### JIT+CUDA P99 Per-Token Latency (ms): 8.23. Result ID: 69f06fe447f34ce924d654e0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"10","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 9.44. Result ID: 69f06fe447f34ce924d654e3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"50","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 11.36. Result ID: 69f06fe447f34ce924d654e6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"100","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 12.54. Result ID: 69f06fe447f34ce924d654e9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"150","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 14.11. Result ID: 69f06fe447f34ce924d654ec. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"200","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 15.65. Result ID: 69f06fe447f34ce924d654ef. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"250","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 16.53. Result ID: 69f06fe447f34ce924d654f4. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"350","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 16.54. Result ID: 69f06fe447f34ce924d654f1. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"300","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 16.54. Result ID: 69f06fe447f34ce924d654f7. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"400","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 16.56. Result ID: 69f06fe447f34ce924d654ee. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"250","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 16.61. Result ID: 69f06fe447f34ce924d654fa. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"450","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 16.65. Result ID: 69f06fe447f34ce924d654fd. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"500","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 16.73. Result ID: 69f06fe447f34ce924d654eb. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"200","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 17.31. Result ID: 69f06fe447f34ce924d654f2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"300","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.3. Result ID: 69f06fe447f34ce924d654e1. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"50","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.38. Result ID: 69f06fe447f34ce924d654fc. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"500","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.39. Result ID: 69f06fe447f34ce924d654de. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"10","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.39. Result ID: 69f06fe447f34ce924d654f9. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"450","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.41. Result ID: 69f06fe447f34ce924d654f6. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"400","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.42. Result ID: 69f06fe447f34ce924d654f3. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"350","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.44. Result ID: 69f06fe447f34ce924d654f0. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"300","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.46. Result ID: 69f06fe447f34ce924d654ed. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"250","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.49. Result ID: 69f06fe447f34ce924d654e4. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"100","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.49. Result ID: 69f06fe447f34ce924d654ea. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"200","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 18.9. Result ID: 69f06fe447f34ce924d654f5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"350","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### HuggingFace P99 Per-Token Latency (ms): 18.94. Result ID: 69f06fe447f34ce924d654e7. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"150","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 20.5. Result ID: 69f06fe447f34ce924d654f8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"400","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 22.09. Result ID: 69f06fe447f34ce924d654fb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"450","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### JIT+CUDA P99 Per-Token Latency (ms): 23.68. Result ID: 69f06fe447f34ce924d654fe. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Generation Length":"500","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 24.03. Result ID: 69f06fe447f34ce924d654e8. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"150","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 31.59. Result ID: 69f06fe447f34ce924d654e5. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"100","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 52.28. Result ID: 69f06fe447f34ce924d654e2. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"50","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. ### TensorRT–LLM P99 Per-Token Latency (ms): 68.84. Result ID: 69f06fe447f34ce924d654df. [Reporting paper]() Recorded evaluation configuration: {"Generation Length":"10","Hardware":"NVIDIA H100","Precision":"FP16","Batch size":"1","Number of trials":"1000"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.