Latency Measurement on LLaMA linear layers 13B (inference)
0.0499Latency (ms)MBOK
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MBOKWeight Size=13824 × 5120, Hardware=A100 GPU2025.05 | 0.0499 | 8.7 | |
| MBOKWeight Size=5120 × 5120, Hardware=A100 GPU2025.05 | 0.0507 | 3.25 | |
| MBOKWeight Size=5120 × 13824, Hardware=A100 GPU2025.05 | 0.051 | 8.4 | |
| FP16Weight Size=5120 × 5120, Hardware=A100 GPU2025.05 | 0.1654 | — | |
| FP16Weight Size=5120 × 13824, Hardware=A100 GPU2025.05 | 0.4283 | — | |
| FP16Weight Size=13824 × 5120, Hardware=A100 GPU2025.05 | 0.4341 | — | |
| QUIP#Weight Size=5120 × 5120, Hardware=A100 GPU2025.05 | 0.6226 | 0.27 | |
| QUIP#Weight Size=5120 × 13824, Hardware=A100 GPU2025.05 | 0.6284 | 0.68 | |
| QUIP#Weight Size=13824 × 5120, Hardware=A100 GPU2025.05 | 0.6284 | 0.69 | |
| QTIPWeight Size=5120 × 5120, Hardware=A100 GPU2025.05 | 1.9637 | 0.08 | |
| QTIPWeight Size=13824 × 5120, Hardware=A100 GPU2025.05 | 5.2119 | 0.08 | |
| QTIPWeight Size=5120 × 13824, Hardware=A100 GPU2025.05 | 5.2368 | 0.09 |