Decoding Throughput on Llama 2 70B v1.0 (inference)
23.5Throughput (TOK/s)QTIP
Evaluation Results
| Method | Links | |
|---|---|---|
| QTIPBITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 23.5 | |
| QUIP#BITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 22.2 | |
| QTIPBITS=3, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 19.1 | |
| QTIPBITS=4, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 16.3 | |
| AQLMBITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06 | 8.78 |