Inference Latency Measurement on LLaMA-13B 5120 × 5120 linear layer
0.0507Latency (ms)MBOK
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MBOKBatch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 51202025.05 | 0.0507 | 3.25 | |
| FP16Batch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 51202025.05 | 0.1654 | — | |
| QUIP#Batch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 51202025.05 | 0.6226 | 0.27 | |
| QTIPBatch Size=1, Hardware=A100 GPU, Library=BitBLAS, Backbone=LLaMA-13B, Weight Size=5120 × 51202025.05 | 1.9637 | 0.08 |