Decoding Throughput on Llama 2 7B inference v1.0
188Decoding Throughput (TOK/s)QTIP
Evaluation Results
| Method | Links | |
|---|---|---|
| QTIPBITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 188 | |
| QUIP#BITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 186 | |
| QTIPBITS=3, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 161 | |
| QTIPBITS=4, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 140 | |
| AQLMBITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 81.5 | |
| FP16BITS=16, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 55.9 |