List your company
ResearchFormerly WizWand

Linear Layer Latency Inference on Llama-3-8B decoder block

153Latency (µs)CodeGEMM
-211.562,249.224,7107,170.78Dec 19, 2025
Updated 8mo ago

Evaluation results

MethodLinks
CodeGEMMBatch Size=1, variant=m1v42025.12153
QUIP#Batch Size=1, variant=e8p2025.12163
CodeGEMMBatch Size=1, variant=m2v82025.12172
QTIPBatch Size=1, variant=r22025.12190
AQLMBatch Size=1, variant=2x82025.12250
cuBLASBatch Size=12025.12332
cuBLASBatch Size=42025.12333
cuBLASBatch Size=82025.12336
cuBLASBatch Size=162025.12340
CodeGEMMBatch Size=4, variant=m1v42025.12405
QUIP#Batch Size=4, variant=e8p2025.12445
CodeGEMMBatch Size=4, variant=m2v82025.12491
QTIPBatch Size=4, variant=r22025.12550
AQLMBatch Size=1, variant=1x162025.12646
CodeGEMMBatch Size=8, variant=m1v42025.12744
AQLMBatch Size=4, variant=2x82025.12794
QUIP#Batch Size=8, variant=e8p2025.12818
CodeGEMMBatch Size=8, variant=m2v82025.12909
DequantBatch Size=12025.121,027
DequantBatch Size=42025.121,027
DequantBatch Size=82025.121,027
DequantBatch Size=162025.121,027
QTIPBatch Size=8, variant=r22025.121,034
cuBLAS+DequantBatch Size=12025.121,360
cuBLAS+DequantBatch Size=42025.121,361
cuBLAS+DequantBatch Size=82025.121,364
cuBLAS+DequantBatch Size=162025.121,367
CodeGEMMBatch Size=16, variant=m1v42025.121,416
AQLMBatch Size=8, variant=2x82025.121,515
QUIP#Batch Size=16, variant=e8p2025.121,554
CodeGEMMBatch Size=16, variant=m2v82025.121,748
QTIPBatch Size=16, variant=r22025.121,991
AQLMBatch Size=4, variant=1x162025.122,373
AQLMBatch Size=16, variant=2x82025.122,959
AQLMBatch Size=8, variant=1x162025.124,695
AQLMBatch Size=16, variant=1x162025.129,267