Inference Latency Measurement on LLaMA-13B (13824 × 5120 linear layer)
0.0499Latency (ms)MBOK
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MBOKBatch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=13824 × 51202025.05 | 0.0499 | 8.7 | |
| FP16Batch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=13824 × 51202025.05 | 0.4341 | — | |
| QUIP#Batch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=13824 × 51202025.05 | 0.6284 | 0.69 | |
| QTIPBatch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=13824 × 51202025.05 | 5.2119 | 0.08 |