End-to-end Serving Throughput and Accuracy on Qwen3-4B-Thinking-2507
66.4Throughput (tokens/s/user)Naive INT2
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Naive INT2Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 66.4 | 1,797.4 | 0 | |
| QuaRot-INT2Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 66.3 | 1,795.8 | 1.4 | |
| OSCARConcurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 63.3 | 1,723.9 | 71.86 | |
| Saw-INT4Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 50.6 | 1,444.1 | 73.11 | |
| BF16Concurrent requests=32, Input tokens=8192, Output tokens=1024, Hardware=1xH100 (80GB)2026.05 | 41.1 | 1,187.9 | 75.64 |