List your company
ResearchFormerly WizWand

LLM Serving on High-load generation workload w/o NVLink

878Time To First Token (ms)DeInfer (w/ low-rank KV cache)
482.563,151.785,8218,490.22Apr 20, 2026
Updated 6mo ago

Evaluation results

MethodLinks
DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.04878154
DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.04906101
DeInferModel=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.0492747
DeInfer (w/ low-rank KV cache)Model=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.04940283
DeInferModel=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.0494446
DeInferModel=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.0496057
DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.041,81288
DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.041,840354
DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.042,085216
DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.042,08891
DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.042,093105
DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.042,102115
DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.042,11088
DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.042,137108
DeInferModel=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.042,192115
DeInfer (w/ low-rank KV cache)Model=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.042,216660
DeInferModel=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.042,24687
DeInfer (w/ low-rank KV cache)Model=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.042,484143
BaseModel=OPT-30B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.043,12654
BaseModel=OPT-30B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.043,56647
BaseModel=OPT-30B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.043,84547
BaseModel=LLaMA-65B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.048,164206
BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=2, NVLink Status=w/o. NVLink2026.048,950156
BaseModel=LLaMA-65B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.049,359103
BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=4, NVLink Status=w/o. NVLink2026.0410,064122
BaseModel=LLaMA-65B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.0410,451101
BaseModel=LLaMA-3-70B, Tensor Parallelism (TP)=8, NVLink Status=w/o. NVLink2026.0410,76496