Prefill Compute Efficiency on Cloud TPU v6e
15.23Prefill MFUMamba2 2.7B
Evaluation Results
| Method | Links | |
|---|---|---|
| Mamba2 2.7BPrompt Length=1024, Model size=2.7B, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 15.23 | |
| Mamba2 2.7BPrompt Length=4096, Model size=2.7B, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 12.96 | |
| Mamba2 1.3BPrompt Length=1024, Model size=1.3B, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 12.53 | |
| Mamba2 1.3BPrompt Length=4096, Model size=1.3B, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 11.67 | |
| Mamba2 780MPrompt Length=4096, Model size=780M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 11.33 | |
| Mamba2 780MPrompt Length=1024, Model size=780M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 10.62 | |
| Mamba2 2.7BPrompt Length=8192, Model size=2.7B, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 9.71 | |
| Mamba2 370MPrompt Length=4096, Model size=370M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 9.04 | |
| Mamba2 1.3BPrompt Length=8192, Model size=1.3B, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 8.39 | |
| Mamba2 130MPrompt Length=4096, Model size=130M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 8.23 | |
| Mamba2 780MPrompt Length=8192, Model size=780M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 8.2 | |
| Mamba2 130MPrompt Length=8192, Model size=130M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 7.68 | |
| Mamba2 370MPrompt Length=8192, Model size=370M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 7.6 | |
| Mamba2 370MPrompt Length=1024, Model size=370M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 7.47 | |
| Mamba2 130MPrompt Length=1024, Model size=130M, Batch size=1, Precision=BF16, Peak TFLOPS=9182026.03 | 6.22 |