LLM Inference on Qwen3-8B (2k prompts Balanced workload)
61,106Throughput (tok/s)v1
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| v1Concurrency (c)=512, Hardware=4x H200, Data Parallelism (DP)=42026.05 | 61,106 | 464 | 14.7 | |
| EB+Concurrency (c)=512, Hardware=4x H200, Data Parallelism (DP)=42026.05 | 60,988 | 413 | 14.9 | |
| 2P+2DConcurrency (c)=512, Hardware=4x H2002026.05 | 48,895 | 1,128 | 17.3 | |
| 1P+3DConcurrency (c)=256, Hardware=4x H2002026.05 | 47,433 | 681 | 8.9 | |
| 1P+3DConcurrency (c)=512, Hardware=4x H2002026.05 | 46,261 | 5,753 | 9.2 | |
| EB+Concurrency (c)=256, Hardware=4x H200, Data Parallelism (DP)=42026.05 | 43,729 | 152 | 10.9 | |
| v1Concurrency (c)=256, Hardware=4x H200, Data Parallelism (DP)=42026.05 | 43,648 | 158 | 10.9 | |
| 2P+2DConcurrency (c)=256, Hardware=4x H2002026.05 | 42,016 | 395 | 10.9 | |
| 1P+3DConcurrency (c)=128, Hardware=4x H2002026.05 | 31,341 | 249 | 7.5 | |
| 2P+2DConcurrency (c)=128, Hardware=4x H2002026.05 | 29,388 | 178 | 8.2 | |
| EB+Concurrency (c)=128, Hardware=4x H200, Data Parallelism (DP)=42026.05 | 28,686 | 74 | 8.6 | |
| v1Concurrency (c)=128, Hardware=4x H200, Data Parallelism (DP)=42026.05 | 28,646 | 75 | 8.6 | |
| 3P+1DConcurrency (c)=256, Hardware=4x H2002026.05 | 26,455 | 596 | 17.6 | |
| 1P+3DConcurrency (c)=512, Hardware=4x RTX PRO 60002026.05 | 25,080 | 4,013 | 29.9 | |
| EB+Concurrency (c)=512, Hardware=4x RTX PRO 6000, Data Parallelism (DP)=42026.05 | 24,037 | 697 | 38.6 | |
| v1Concurrency (c)=512, Hardware=4x RTX PRO 6000, Data Parallelism (DP)=42026.05 | 23,914 | 697 | 38.7 | |
| 3P+1DConcurrency (c)=128, Hardware=4x H2002026.05 | 21,963 | 233 | 11 | |
| 2P+2DConcurrency (c)=512, Hardware=4x RTX PRO 60002026.05 | 20,397 | 1,676 | 43.4 | |
| 1P+3DConcurrency (c)=256, Hardware=4x RTX PRO 60002026.05 | 19,831 | 844 | 22.9 | |
| 2P+2DConcurrency (c)=256, Hardware=4x RTX PRO 60002026.05 | 17,551 | 505 | 26.9 | |
| v1Concurrency (c)=256, Hardware=4x RTX PRO 6000, Data Parallelism (DP)=42026.05 | 16,913 | 235 | 28.6 | |
| EB+Concurrency (c)=256, Hardware=4x RTX PRO 6000, Data Parallelism (DP)=42026.05 | 16,870 | 229 | 28.7 | |
| 1P+3DConcurrency (c)=128, Hardware=4x RTX PRO 60002026.05 | 12,629 | 316 | 19.2 | |
| 2P+2DConcurrency (c)=128, Hardware=4x RTX PRO 60002026.05 | 11,640 | 229 | 21.1 | |
| 3P+1DConcurrency (c)=256, Hardware=4x RTX PRO 60002026.05 | 11,402 | 664 | 42.1 | |
| EB+Concurrency (c)=128, Hardware=4x RTX PRO 6000, Data Parallelism (DP)=42026.05 | 11,063 | 116 | 22.5 | |
| v1Concurrency (c)=128, Hardware=4x RTX PRO 6000, Data Parallelism (DP)=42026.05 | 11,013 | 121 | 22.6 | |
| 3P+1DConcurrency (c)=128, Hardware=4x RTX PRO 60002026.05 | 9,023 | 272 | 27.3 |