End-to-end Serving Throughput and Accuracy on Qwen3-8B
54.2User Throughput (tokens/s)Naive INT2
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Naive INT2Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 54.2 | 1,415.1 | 0 | |
| QuaRot-INT2Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 54.1 | 1,411.9 | 10.14 | |
| OSCARConcurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 52.5 | 1,374 | 69.42 | |
| Saw-INT4Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 43 | 1,183.2 | 69.97 | |
| BF16Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 35.8 | 999.9 | 70.84 |