Inference Latency on A100 GPU
25.4Latency (ms)AWQ
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AWQModel family=LLaMA-2, Size=7B, Bits=32024.06 | 25.4 | — | |
| AWQModel family=LLaMA-2, Size=7B, Bits=22024.06 | 25.4 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=7B, Bits=3, Optimization=Lat.2024.06 | 26.7 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=7B, Bits=2.2, Optimization=Mixed2024.06 | 27.2 | — | |
| LUT-GEMMModel family=LLaMA-2, Size=7B, Bits=32024.06 | 27.4 | — | |
| LUT-GEMMModel family=LLaMA-2, Size=7B, Bits=22024.06 | 27.5 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=7B, Bits=2, Optimization=Lat.2024.06 | 27.7 | — | |
| AWQModel family=LLaMA-3, Size=8B, Bits=32024.06 | 28.5 | — | |
| AWQModel family=LLaMA-3, Size=8B, Bits=22024.06 | 28.5 | — | |
| ShiftAddLLMModel family=LLaMA-3, Size=8B, Bits=2.2, Optimization=Mixed2024.06 | 30.1 | — | |
| OPTQModel family=LLaMA-2, Size=7B, Bits=32024.06 | 31.1 | — | |
| ShiftAddLLMModel family=LLaMA-3, Size=8B, Bits=3, Optimization=Lat.2024.06 | 31.4 | — | |
| LUT-GEMMModel family=LLaMA-3, Size=8B, Bits=32024.06 | 31.7 | — | |
| LUT-GEMMModel family=LLaMA-3, Size=8B, Bits=22024.06 | 31.7 | — | |
| AWQModel family=LLaMA-2, Size=13B, Bits=32024.06 | 31.8 | — | |
| AWQModel family=LLaMA-2, Size=13B, Bits=22024.06 | 31.8 | — | |
| ShiftAddLLMModel family=LLaMA-3, Size=8B, Bits=2, Optimization=Lat.2024.06 | 31.9 | — | |
| FP16Model family=LLaMA-2, Size=7B, Bits=162024.06 | 32.6 | — | |
| LUT-GEMMModel family=LLaMA-2, Size=13B, Bits=22024.06 | 33.3 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=13B, Bits=3, Optimization=Lat.2024.06 | 33.8 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=13B, Bits=2, Optimization=Lat.2024.06 | 33.9 | — | |
| OPTQModel family=LLaMA-2, Size=7B, Bits=22024.06 | 34.2 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=13B, Bits=2.2, Optimization=Mixed2024.06 | 34.3 | — | |
| LUT-GEMMModel family=LLaMA-2, Size=13B, Bits=32024.06 | 34.7 | — | |
| OPTQModel family=LLaMA-3, Size=8B, Bits=32024.06 | 36.2 | — | |
| OPTQModel family=LLaMA-3, Size=8B, Bits=22024.06 | 36.8 | — | |
| FP16Model family=LLaMA-3, Size=8B, Bits=162024.06 | 38.8 | — | |
| OPTQModel family=LLaMA-2, Size=13B, Bits=22024.06 | 38.8 | — | |
| OPTQModel family=LLaMA-2, Size=13B, Bits=32024.06 | 42.2 | — | |
| FP16Model family=LLaMA-2, Size=13B, Bits=162024.06 | 43.1 | — | |
| AWQModel family=LLaMA-3, Size=70B, Bits=32024.06 | 67.7 | — | |
| AWQModel family=LLaMA-3, Size=70B, Bits=22024.06 | 67.7 | — | |
| AWQModel family=LLaMA-2, Size=70B, Bits=32024.06 | 68 | — | |
| AWQModel family=LLaMA-2, Size=70B, Bits=22024.06 | 68 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=70B, Bits=3, Optimization=Lat.2024.06 | 70.9 | — | |
| LUT-GEMMModel family=LLaMA-2, Size=70B, Bits=22024.06 | 71 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=70B, Bits=2, Optimization=Lat.2024.06 | 72.1 | — | |
| LUT-GEMMModel family=LLaMA-2, Size=70B, Bits=32024.06 | 72.6 | — | |
| ShiftAddLLMModel family=LLaMA-3, Size=70B, Bits=3, Optimization=Lat.2024.06 | 72.9 | — | |
| ShiftAddLLMModel family=LLaMA-2, Size=70B, Bits=2.2, Optimization=Mixed2024.06 | 75.1 | — | |
| ShiftAddLLMModel family=LLaMA-3, Size=70B, Bits=2.2, Optimization=Mixed2024.06 | 76.4 | — | |
| LUT-GEMMModel family=LLaMA-3, Size=70B, Bits=22024.06 | 77.2 | — | |
| LUT-GEMMModel family=LLaMA-3, Size=70B, Bits=32024.06 | 77.5 | — | |
| ShiftAddLLMModel family=LLaMA-3, Size=70B, Bits=2, Optimization=Lat.2024.06 | 78.3 | — | |
| OPTQModel family=LLaMA-2, Size=70B, Bits=32024.06 | 81.9 | — | |
| OPTQModel family=LLaMA-2, Size=70B, Bits=22024.06 | 82.5 | — | |
| OPTQModel family=LLaMA-3, Size=70B, Bits=32024.06 | 90.7 | — | |
| OPTQModel family=LLaMA-3, Size=70B, Bits=22024.06 | 91.2 | — | |
| Cosmos PolicyTrainable Params=2.0B2026.06 | — | 0.7 | |
| GR00T-N1.7Trainable Params=1.6B2026.06 | — | 6.5 | |
| MotionWAMTrainable Params=2.5B2026.06 | — | 4.9 | |
| Qwen3DiTTrainable Params=2.3B2026.06 | — | 9 |