Inference Latency on OPT model family
6.2Latency (ms)AWQ
Evaluation results
| Method | Links | |
|---|---|---|
| AWQBits=3, Model Size=125M, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 6.2 | |
| LUT-GEMMBits=2, Model Size=125M, Hardware=A100 GPU2024.06 | 6.2 | |
| AWQBits=2, Model Size=125M, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 6.2 | |
| LUT-GEMMBits=3, Model Size=125M, Hardware=A100 GPU2024.06 | 6.3 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=125M, Hardware=A100 GPU2024.06 | 6.3 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=125M, Hardware=A100 GPU2024.06 | 6.3 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=125M, Hardware=A100 GPU2024.06 | 6.4 | |
| FP16Bits=16, Model Size=125M, Hardware=A100 GPU2024.06 | 7.8 | |
| OPTQBits=2, Model Size=125M, Hardware=A100 GPU2024.06 | 8.2 | |
| OPTQBits=3, Model Size=125M, Hardware=A100 GPU2024.06 | 8.3 | |
| LUT-GEMMBits=3, Model Size=350M, Hardware=A100 GPU2024.06 | 11.7 | |
| LUT-GEMMBits=2, Model Size=350M, Hardware=A100 GPU2024.06 | 11.8 | |
| LUT-GEMMBits=2, Model Size=1.3B, Hardware=A100 GPU2024.06 | 11.8 | |
| AWQBits=3, Model Size=350M, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 12.1 | |
| AWQBits=2, Model Size=350M, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 12.1 | |
| AWQBits=3, Model Size=1.3B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 12.3 | |
| AWQBits=2, Model Size=1.3B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 12.3 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=1.3B, Hardware=A100 GPU2024.06 | 12.3 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=350M, Hardware=A100 GPU2024.06 | 12.4 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=1.3B, Hardware=A100 GPU2024.06 | 12.5 | |
| LUT-GEMMBits=3, Model Size=1.3B, Hardware=A100 GPU2024.06 | 12.6 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=1.3B, Hardware=A100 GPU2024.06 | 12.6 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=350M, Hardware=A100 GPU2024.06 | 12.6 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=350M, Hardware=A100 GPU2024.06 | 13.3 | |
| OPTQBits=3, Model Size=1.3B, Hardware=A100 GPU2024.06 | 15 | |
| FP16Bits=16, Model Size=350M, Hardware=A100 GPU2024.06 | 15.1 | |
| LUT-GEMMBits=3, Model Size=2.7B, Hardware=A100 GPU2024.06 | 15.5 | |
| LUT-GEMMBits=2, Model Size=2.7B, Hardware=A100 GPU2024.06 | 15.7 | |
| LUT-GEMMBits=2, Model Size=6.7B, Hardware=A100 GPU2024.06 | 15.7 | |
| OPTQBits=3, Model Size=350M, Hardware=A100 GPU2024.06 | 15.9 | |
| OPTQBits=2, Model Size=1.3B, Hardware=A100 GPU2024.06 | 15.9 | |
| OPTQBits=2, Model Size=350M, Hardware=A100 GPU2024.06 | 16.1 | |
| AWQBits=3, Model Size=2.7B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 16.3 | |
| AWQBits=3, Model Size=6.7B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 16.3 | |
| AWQBits=2, Model Size=2.7B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 16.3 | |
| AWQBits=2, Model Size=6.7B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 16.3 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=2.7B, Hardware=A100 GPU2024.06 | 16.6 | |
| FP16Bits=16, Model Size=1.3B, Hardware=A100 GPU2024.06 | 16.7 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=6.7B, Hardware=A100 GPU2024.06 | 16.7 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=2.7B, Hardware=A100 GPU2024.06 | 16.8 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=6.7B, Hardware=A100 GPU2024.06 | 16.9 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=2.7B, Hardware=A100 GPU2024.06 | 16.9 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=6.7B, Hardware=A100 GPU2024.06 | 16.9 | |
| LUT-GEMMBits=3, Model Size=6.7B, Hardware=A100 GPU2024.06 | 17 | |
| LUT-GEMMBits=3, Model Size=13B, Hardware=A100 GPU2024.06 | 19.5 | |
| OPTQBits=2, Model Size=2.7B, Hardware=A100 GPU2024.06 | 19.7 | |
| LUT-GEMMBits=2, Model Size=13B, Hardware=A100 GPU2024.06 | 19.8 | |
| OPTQBits=2, Model Size=6.7B, Hardware=A100 GPU2024.06 | 19.9 | |
| AWQBits=3, Model Size=13B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 20 | |
| AWQBits=2, Model Size=13B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 20 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=13B, Hardware=A100 GPU2024.06 | 20.8 | |
| FP16Bits=16, Model Size=2.7B, Hardware=A100 GPU2024.06 | 20.9 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=13B, Hardware=A100 GPU2024.06 | 20.9 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=13B, Hardware=A100 GPU2024.06 | 21 | |
| OPTQBits=3, Model Size=6.7B, Hardware=A100 GPU2024.06 | 21.1 | |
| OPTQBits=3, Model Size=2.7B, Hardware=A100 GPU2024.06 | 21.5 | |
| FP16Bits=16, Model Size=6.7B, Hardware=A100 GPU2024.06 | 22.2 | |
| LUT-GEMMBits=2, Model Size=30B, Hardware=A100 GPU2024.06 | 23.6 | |
| LUT-GEMMBits=3, Model Size=30B, Hardware=A100 GPU2024.06 | 23.7 | |
| AWQBits=3, Model Size=30B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 24.5 | |
| AWQBits=2, Model Size=30B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 24.5 | |
| OPTQBits=2, Model Size=13B, Hardware=A100 GPU2024.06 | 24.7 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=30B, Hardware=A100 GPU2024.06 | 25.4 | |
| OPTQBits=3, Model Size=13B, Hardware=A100 GPU2024.06 | 26.4 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=30B, Hardware=A100 GPU2024.06 | 27.1 | |
| FP16Bits=16, Model Size=13B, Hardware=A100 GPU2024.06 | 29.5 | |
| OPTQBits=3, Model Size=30B, Hardware=A100 GPU2024.06 | 30.1 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=30B, Hardware=A100 GPU2024.06 | 30.7 | |
| OPTQBits=2, Model Size=30B, Hardware=A100 GPU2024.06 | 31.5 | |
| LUT-GEMMBits=2, Model Size=66B, Hardware=A100 GPU2024.06 | 33.2 | |
| LUT-GEMMBits=3, Model Size=66B, Hardware=A100 GPU2024.06 | 39.5 | |
| AWQBits=3, Model Size=66B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 40.9 | |
| AWQBits=2, Model Size=66B, Hardware=A100 GPU, Kernel=INT4 kernel2024.06 | 40.9 | |
| ShiftAddLLM (Lat.)Bits=2, Model Size=66B, Hardware=A100 GPU2024.06 | 42.9 | |
| ShiftAddLLM (Mixed)Bits=2.2, Model Size=66B, Hardware=A100 GPU2024.06 | 45.7 | |
| OPTQBits=2, Model Size=66B, Hardware=A100 GPU2024.06 | 50.4 | |
| OPTQBits=3, Model Size=66B, Hardware=A100 GPU2024.06 | 51.5 | |
| FP16Bits=16, Model Size=30B, Hardware=A100 GPU2024.06 | 51.7 | |
| ShiftAddLLM (Lat.)Bits=3, Model Size=66B, Hardware=A100 GPU2024.06 | 54.1 |