Inference Latency Measurement on LLaMA 5120 × 13824 linear layer 13B
0.051Latency (ms)MBOK
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MBOKBatch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 138242025.05 | 0.051 | 8.4 | |
| FP16Batch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 138242025.05 | 0.4283 | — | |
| QUIP#Batch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 138242025.05 | 0.6284 | 0.68 | |
| QTIPBatch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 138242025.05 | 5.2368 | 0.09 |