Inference Latency on OPT-175B first FFN layer
0.225Latency (ms)LUT-GEMM
Evaluation results
| Method | Links | |
|---|---|---|
| LUT-GEMMSchemes=UQ, BCQ, # Bits=3, Hardware=A100-80GB-GPU2025.06 | 0.225 | |
| LUT-GEMMSchemes=UQ, BCQ, # Bits=4, Hardware=A100-80GB-GPU2025.06 | 0.2688 | |
| AWQSchemes=UQ, # Bits=4, Hardware=A100-80GB-GPU2025.06 | 0.3238 | |
| GPTQSchemes=UQ, # Bits=3, Hardware=A100-80GB-GPU2025.06 | 0.3599 | |
| cuBLAS# Bits=16, Hardware=A100-80GB-GPU2025.06 | 0.7256 |