LLM Inference Performance on LLaMA-3 8B
56.03TTFT (ms)MXFP4
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| MXFP4Bsz=1, Input sequence length=2k tokens2026.03 | 56.03 | 2.36 | 6.94 | 2.51 | 25.48 | 0.72 | |
| SERQ-MXFP4Bsz=1, Input sequence length=2k tokens2026.03 | 62.31 | 2.12 | 7.03 | 2.48 | 39.28 | 0.47 | |
| FP16Bsz=1, Input sequence length=2k tokens2026.03 | 132.38 | 1 | 17.44 | 1 | 18.44 | 1 | |
| only-MXFP4Bsz=8, Input sequence length=2k tokens2026.03 | 542.58 | 2.31 | 17.4 | 1.6 | 54.49 | 1.14 | |
| SERQ-MXFP4Bsz=8, Input sequence length=2k tokens2026.03 | 587.17 | 2.13 | 17.48 | 1.6 | 57.08 | 1.08 | |
| only-MXFP4Bsz=16, Input sequence length=2k tokens2026.03 | 1,113.06 | 2.26 | 29.35 | 1.36 | 100.71 | 1.13 | |
| SERQ-MXFP4Bsz=16, Input sequence length=2k tokens2026.03 | 1,206.69 | 2.08 | 29.44 | 1.35 | 103.27 | 1.11 | |
| FP16Bsz=8, Input sequence length=2k tokens2026.03 | 1,252.24 | 1 | 27.9 | 1 | 61.93 | 1 | |
| only-MXFP4Bsz=32, Input sequence length=2k tokens2026.03 | 2,257.54 | 2.21 | 53.25 | 1.2 | 193.7 | 1.13 | |
| SERQ-MXFP4Bsz=32, Input sequence length=2k tokens2026.03 | 2,453.29 | 2.03 | 53.34 | 1.2 | 202.13 | 1.08 | |
| FP16Bsz=16, Input sequence length=2k tokens2026.03 | 2,515.08 | 1 | 39.85 | 1 | 114.24 | 1 | |
| FP16Bsz=32, Input sequence length=2k tokens2026.03 | 4,987.05 | 1 | 63.76 | 1 | 218.01 | 1 |