List your company
ResearchFormerly WizWand

Decoding Throughput on Llama 2 7B inference v1.0

188Decoding Throughput (TOK/s)QTIP
50.61686.283121.95157.617Jun 17, 2024
Updated 8mo ago

Evaluation results

MethodLinks
QTIPBITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06188
QUIP#BITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06186
QTIPBITS=3, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06161
QTIPBITS=4, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.06140
AQLMBITS=2, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.0681.5
FP16BITS=16, Batch size=1, Hardware=RTX6000 Ada (960GB/s mem. BW)2024.0655.9