Large Language Model Inference on Qwen2.5-7B (test)
37.29ThroughputTAQ-IS
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| TAQ-ISBatch size=1, Hardware=NVIDIA A402025.11 | 37.29 | 26.82 | 35 | 26 | |
| TAQ-KLBatch size=1, Hardware=NVIDIA A402025.11 | 32.83 | 30.48 | 19 | 16 | |
| UNIFORM4Batch size=1, Hardware=NVIDIA A402025.11 | 28.71 | 34.83 | 4 | 4 | |
| TAQ-OBatch size=1, Hardware=NVIDIA A402025.11 | 27.86 | 35.93 | 1 | 1 | |
| FP16Batch size=1, Hardware=NVIDIA A402025.11 | 27.63 | 36.25 | — | — | |
| AWQ-Int4Batch size=1, Hardware=NVIDIA A402025.11 | 16.12 | 62.51 | 42 | 72 | |
| GPTQ-Int4Batch size=1, Hardware=NVIDIA A402025.11 | 4.4 | 227.26 | 84 | 527 |