LLM Inference on LLaMA2 7B
11.09TTFT (ms)JIT+CUDA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| JIT+CUDAPrompt Size=100, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 11.09 | — | — | — | |
| JIT+CUDAPrompt Size=150, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 11.17 | — | — | — | |
| JIT+CUDAPrompt Size=200, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 12.7 | — | — | — | |
| JIT+CUDAPrompt Size=50, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 12.95 | — | — | — | |
| JIT+CUDAPrompt Size=10, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 13.36 | — | — | — | |
| JIT+CUDAPrompt Size=300, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 14.91 | — | — | — | |
| JIT+CUDAPrompt Size=350, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 15.17 | — | — | — | |
| JIT+CUDAPrompt Size=250, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 15.53 | — | — | — | |
| TensorRT-LLMPrompt Size=10, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 16 | — | — | — | |
| JIT+CUDAPrompt Size=450, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 17.02 | — | — | — | |
| JIT+CUDAPrompt Size=400, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 17.47 | — | — | — | |
| JIT+CUDAPrompt Size=500, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 17.79 | — | — | — | |
| TensorRT-LLMPrompt Size=50, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 22 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=10, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 24.65 | — | — | — | |
| TensorRT-LLMPrompt Size=100, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 28 | — | — | — | |
| TensorRT-LLMPrompt Size=150, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 29 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=100, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 29.75 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=150, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 31.97 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=50, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 33.66 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=200, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 34.01 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=250, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 35.54 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=350, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 41.03 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=300, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 43.27 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=450, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 46.18 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=400, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 46.47 | — | — | — | |
| TensorRT-LLMPrompt Size=200, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 48 | — | — | — | |
| HuggingFace (PyTorch Eager)Prompt Size=500, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 48.96 | — | — | — | |
| TensorRT-LLMPrompt Size=300, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 68 | — | — | — | |
| TensorRT-LLMPrompt Size=350, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 68 | — | — | — | |
| TensorRT-LLMPrompt Size=250, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 69 | — | — | — | |
| TensorRT-LLMPrompt Size=400, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 86 | — | — | — | |
| TensorRT-LLMPrompt Size=450, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 86 | — | — | — | |
| TensorRT-LLMPrompt Size=500, Hardware=NVIDIA H100, Precision=FP16, Batch Size=12026.04 | 88 | — | — | — | |
| Baseline2026.03 | — | — | 1,052.24 | — | |
| FaR2026.03 | — | 3.04 | 6,075.9 | 477.4 | |
| RADAR2026.03 | — | 50 | 1,937.38 | 84.1 | |
| RoR2026.03 | — | 0.31 | 1,148 | 9.1 |