Inference Latency on OPT-30B
15.7Latency (ms)LUT-GEMM
Evaluation results
| Method | Links | |
|---|---|---|
| LUT-GEMMKernel=LUT-GEMM, bit width (k)=3, group size (g)=64, Hardware=A100-80GB-GPU2025.06 | 15.7 | |
| LUT-GEMMKernel=LUT-GEMM, bit width (k)=3, group size (g)=32, Hardware=A100-80GB-GPU2025.06 | 16.7 | |
| LUT-GEMMKernel=LUT-GEMM, bit width (k)=4, group size (g)=64, Hardware=A100-80GB-GPU2025.06 | 17.8 | |
| LUT-GEMMKernel=LUT-GEMM, bit width (k)=4, group size (g)=32, Hardware=A100-80GB-GPU2025.06 | 18.5 | |
| cuBLASKernel=cuBLAS, bit width (k)=16, group size (g)=N/A, Hardware=A100-80GB-GPU2025.06 | 40.5 |