Linear Layer Latency Inference on Llama-3-8B decoder block
153Latency (µs)CodeGEMM
Evaluation Results
| Method | Links | |
|---|---|---|
| CodeGEMMBatch Size=1, variant=m1v42025.12 | 153 | |
| QUIP#Batch Size=1, variant=e8p2025.12 | 163 | |
| CodeGEMMBatch Size=1, variant=m2v82025.12 | 172 | |
| QTIPBatch Size=1, variant=r22025.12 | 190 | |
| AQLMBatch Size=1, variant=2x82025.12 | 250 | |
| cuBLASBatch Size=12025.12 | 332 | |
| cuBLASBatch Size=42025.12 | 333 | |
| cuBLASBatch Size=82025.12 | 336 | |
| cuBLASBatch Size=162025.12 | 340 | |
| CodeGEMMBatch Size=4, variant=m1v42025.12 | 405 | |
| QUIP#Batch Size=4, variant=e8p2025.12 | 445 | |
| CodeGEMMBatch Size=4, variant=m2v82025.12 | 491 | |
| QTIPBatch Size=4, variant=r22025.12 | 550 | |
| AQLMBatch Size=1, variant=1x162025.12 | 646 | |
| CodeGEMMBatch Size=8, variant=m1v42025.12 | 744 | |
| AQLMBatch Size=4, variant=2x82025.12 | 794 | |
| QUIP#Batch Size=8, variant=e8p2025.12 | 818 | |
| CodeGEMMBatch Size=8, variant=m2v82025.12 | 909 | |
| DequantBatch Size=12025.12 | 1,027 | |
| DequantBatch Size=42025.12 | 1,027 | |
| DequantBatch Size=82025.12 | 1,027 | |
| DequantBatch Size=162025.12 | 1,027 | |
| QTIPBatch Size=8, variant=r22025.12 | 1,034 | |
| cuBLAS+DequantBatch Size=12025.12 | 1,360 | |
| cuBLAS+DequantBatch Size=42025.12 | 1,361 | |
| cuBLAS+DequantBatch Size=82025.12 | 1,364 | |
| cuBLAS+DequantBatch Size=162025.12 | 1,367 | |
| CodeGEMMBatch Size=16, variant=m1v42025.12 | 1,416 | |
| AQLMBatch Size=8, variant=2x82025.12 | 1,515 | |
| QUIP#Batch Size=16, variant=e8p2025.12 | 1,554 | |
| CodeGEMMBatch Size=16, variant=m2v82025.12 | 1,748 | |
| QTIPBatch Size=16, variant=r22025.12 | 1,991 | |
| AQLMBatch Size=4, variant=1x162025.12 | 2,373 | |
| AQLMBatch Size=16, variant=2x82025.12 | 2,959 | |
| AQLMBatch Size=8, variant=1x162025.12 | 4,695 | |
| AQLMBatch Size=16, variant=1x162025.12 | 9,267 |