LLM Serving on High-load generation workload w/o NVLink
878Time To First Token (ms)DeInfer (w/ low-rank KV cache)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 878 | 154 | |
| DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 906 | 101 | |
| DeInferModel=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 927 | 47 | |
| DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 940 | 283 | |
| DeInferModel=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 944 | 46 | |
| DeInferModel=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 960 | 57 | |
| DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 1,812 | 88 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 1,840 | 354 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 2,085 | 216 | |
| DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 2,088 | 91 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 2,093 | 105 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 2,102 | 115 | |
| DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 2,110 | 88 | |
| DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 2,137 | 108 | |
| DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 2,192 | 115 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 2,216 | 660 | |
| DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 2,246 | 87 | |
| DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 2,484 | 143 | |
| BaseModel=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 3,126 | 54 | |
| BaseModel=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 3,566 | 47 | |
| BaseModel=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 3,845 | 47 | |
| BaseModel=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 8,164 | 206 | |
| BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04 | 8,950 | 156 | |
| BaseModel=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 9,359 | 103 | |
| BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04 | 10,064 | 122 | |
| BaseModel=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 10,451 | 101 | |
| BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04 | 10,764 | 96 |