# LLM Inference on LLaMA2 7B Source: [LLM Inference on LLaMA2 7B]() Collection index: [llms.txt]() Data updated: 2026-04-28T08:29:36.302Z Evaluation of the inference performance and resource overhead metrics, including Extra Storage and Latency Overhead, for the LLaMA2 7B model. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | LLM Inference | | Dataset (including any stated split) | LLaMA2 7B | | Metric | TTFT (ms) | | Unit | TIME | | Direction | Lower is better | | Results loaded | 37 of 37 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [TTFT (ms)](): Lower is better; unit: TIME. - [Extra Storage](): Lower is better; unit: PERCENT. - [Latency (ms)](): Lower is better; unit: TIME. - [Latency Overhead](): Lower is better; unit: PERCENT. ## Recorded results | Method | TTFT (ms) | Result date | Report paper | | --- | --- | --- | --- | | JIT+CUDA | 11.09 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 11.17 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 12.7 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 12.95 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 13.36 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 14.91 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 15.17 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 15.53 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 16 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 17.02 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 17.47 | Not recorded | 69f03f4455a3c32af89fa244 | | JIT+CUDA | 17.79 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 22 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 24.65 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 28 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 29 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 29.75 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 31.97 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 33.66 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 34.01 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 35.54 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 41.03 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 43.27 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 46.18 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 46.47 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 48 | Not recorded | 69f03f4455a3c32af89fa244 | | HuggingFace (PyTorch Eager) | 48.96 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 68 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 68 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 69 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 86 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 86 | Not recorded | 69f03f4455a3c32af89fa244 | | TensorRT-LLM | 88 | Not recorded | 69f03f4455a3c32af89fa244 | ### JIT+CUDA TTFT (ms): 11.09. Result ID: 69f06fe447f34ce924d654c5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"100","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 11.17. Result ID: 69f06fe447f34ce924d654c8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"150","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 12.7. Result ID: 69f06fe447f34ce924d654cb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"200","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 12.95. Result ID: 69f06fe447f34ce924d654c2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"50","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 13.36. Result ID: 69f06fe447f34ce924d654bf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"10","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 14.91. Result ID: 69f06fe447f34ce924d654d1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"300","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 15.17. Result ID: 69f06fe447f34ce924d654d4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"350","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 15.53. Result ID: 69f06fe447f34ce924d654ce. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"250","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 16. Result ID: 69f06fe447f34ce924d654be. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"10","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 17.02. Result ID: 69f06fe447f34ce924d654da. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"450","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 17.47. Result ID: 69f06fe447f34ce924d654d7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"400","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### JIT+CUDA TTFT (ms): 17.79. Result ID: 69f06fe447f34ce924d654dd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Prompt Size":"500","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 22. Result ID: 69f06fe447f34ce924d654c1. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"50","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 24.65. Result ID: 69f06fe447f34ce924d654bd. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"10","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 28. Result ID: 69f06fe447f34ce924d654c4. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"100","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 29. Result ID: 69f06fe447f34ce924d654c7. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"150","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 29.75. Result ID: 69f06fe447f34ce924d654c3. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"100","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 31.97. Result ID: 69f06fe447f34ce924d654c6. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"150","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 33.66. Result ID: 69f06fe447f34ce924d654c0. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"50","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 34.01. Result ID: 69f06fe447f34ce924d654c9. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"200","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 35.54. Result ID: 69f06fe447f34ce924d654cc. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"250","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 41.03. Result ID: 69f06fe447f34ce924d654d2. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"350","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 43.27. Result ID: 69f06fe447f34ce924d654cf. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"300","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 46.18. Result ID: 69f06fe447f34ce924d654d8. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"450","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 46.47. Result ID: 69f06fe447f34ce924d654d5. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"400","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 48. Result ID: 69f06fe447f34ce924d654ca. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"200","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### HuggingFace (PyTorch Eager) TTFT (ms): 48.96. Result ID: 69f06fe447f34ce924d654db. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"500","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 68. Result ID: 69f06fe447f34ce924d654d0. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"300","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 68. Result ID: 69f06fe447f34ce924d654d3. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"350","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 69. Result ID: 69f06fe447f34ce924d654cd. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"250","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 86. Result ID: 69f06fe447f34ce924d654d6. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"400","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 86. Result ID: 69f06fe447f34ce924d654d9. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"450","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. ### TensorRT-LLM TTFT (ms): 88. Result ID: 69f06fe447f34ce924d654dc. [Reporting paper]() Recorded evaluation configuration: {"Prompt Size":"500","Hardware":"NVIDIA H100","Precision":"FP16","Batch Size":"1"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.