Inference Efficiency on Qwen2.5-7B
1,480.2Throughput (tokens/s)AWQ
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| AWQQuantization=4-bit2026.02 | 1,480.2 | 0.675 | — | |
| DACQ HybridQuantization=4-bit2026.02 | 1,093.1 | 0.915 | — | |
| DACQ LogisticQuantization=4-bit2026.02 | 1,035.4 | 0.966 | — | |
| UnquantizedQuantization=Unquantized2026.02 | 929.4 | 1.0759 | — | |
| FineRMoE#Param (B)=26.65, #A-Param (B)=7.94, #Generated Tokens=1102026.03 | 27.3 | — | 178.3 | |
| DU#Param (B)=47.54, #A-Param (B)=13.32, #Generated Tokens=320002026.03 | 25.6 | — | 84.8 | |
| S16A4#Param (B)=7.62, #A-Param (B)=3.34, #Generated Tokens=472026.03 | 24 | — | 78.5 | |
| NVShard#Param (B)=47.55, #A-Param (B)=7.63, #Generated Tokens=3322026.03 | 18.9 | — | 137.8 | |
| C32A2#Param (B)=184.42, #A-Param (B)=13.33, #Generated Tokens=1072026.03 | 0.2 | — | 50,245.9 |