List your company
ResearchFormerly WizWand

P99 Per-Token Latency on LLaMA-2 7B (inference)

8.23P99 Per-Token Latency (ms)JIT+CUDA
5.805622.170338.53554.8997Apr 25, 2026
Updated 6mo ago

Evaluation results

MethodLinks
JIT+CUDAGeneration Length=10, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.048.23
JIT+CUDAGeneration Length=50, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.049.44
JIT+CUDAGeneration Length=100, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0411.36
JIT+CUDAGeneration Length=150, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0412.54
JIT+CUDAGeneration Length=200, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0414.11
JIT+CUDAGeneration Length=250, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0415.65
TensorRT–LLMGeneration Length=350, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0416.53
TensorRT–LLMGeneration Length=300, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0416.54
TensorRT–LLMGeneration Length=400, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0416.54
TensorRT–LLMGeneration Length=250, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0416.56
TensorRT–LLMGeneration Length=450, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0416.61
TensorRT–LLMGeneration Length=500, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0416.65
TensorRT–LLMGeneration Length=200, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0416.73
JIT+CUDAGeneration Length=300, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0417.31
HuggingFaceGeneration Length=50, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.3
HuggingFaceGeneration Length=500, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.38
HuggingFaceGeneration Length=10, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.39
HuggingFaceGeneration Length=450, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.39
HuggingFaceGeneration Length=400, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.41
HuggingFaceGeneration Length=350, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.42
HuggingFaceGeneration Length=300, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.44
HuggingFaceGeneration Length=250, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.46
HuggingFaceGeneration Length=100, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.49
HuggingFaceGeneration Length=200, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.49
JIT+CUDAGeneration Length=350, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.9
HuggingFaceGeneration Length=150, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0418.94
JIT+CUDAGeneration Length=400, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0420.5
JIT+CUDAGeneration Length=450, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0422.09
JIT+CUDAGeneration Length=500, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0423.68
TensorRT–LLMGeneration Length=150, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0424.03
TensorRT–LLMGeneration Length=100, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0431.59
TensorRT–LLMGeneration Length=50, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0452.28
TensorRT–LLMGeneration Length=10, Hardware=NVIDIA H100, Precision=FP16, Batch size=1, Number of trials=10002026.0468.84