LLM Serving on High-load generation workload w/ NVLink
443TTFT (ms)DeInfer
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DeInferModel=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 443 | 33 | |
| DeInferModel=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 448 | 37 | |
| DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 457 | 86 | |
| DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 514 | 146 | |
| DeInferModel=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 581 | 50 | |
| DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 640 | 276 | |
| BaseModel=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 649 | 41 | |
| BaseModel=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 654 | 42 | |
| BaseModel=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 724 | 50 | |
| DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 749 | 56 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 782 | 184 | |
| DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 797 | 78 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 817 | 93 | |
| DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 875 | 64 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 901 | 330 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 948 | 96 | |
| DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 983 | 76 | |
| DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 1,162 | 92 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 1,214 | 642 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 1,377 | 127 | |
| BaseModel=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 1,417 | 98 | |
| BaseModel=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 1,461 | 94 | |
| DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 1,463 | 90 | |
| BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/ NVLink2026.04 | 1,499 | 115 | |
| BaseModel=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 1,525 | 191 | |
| BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/ NVLink2026.04 | 1,605 | 93 | |
| BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/ NVLink2026.04 | 1,662 | 156 |