Decode-phase Efficiency Benchmarking on H100-80GB GPU
13Peak Memory (MB)Baseline
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| BaselineSize=Base, Params=757M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 13 | 1,221 | 13 | |
| BaselineSize=Medium, Params=1.7B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 25 | 2,075 | 7,896 | |
| BaselineSize=Large, Params=2.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 33 | 3,061 | 5,353 | |
| BaselineSize=XL, Params=3.9B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 49 | 4,544 | 3,605 | |
| BaselineSize=XXL, Params=5.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 62 | 5,920 | 2,767 | |
| MHAHadSize=Tiny, Params=117M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 3,454 | 275 | 59,478 | |
| BaselineSize=Tiny, Params=124M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 3,490 | 272 | 60 | |
| MHAHadSize=Small, Params=329M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 8,525 | 635 | 25,797 | |
| BaselineSize=Small, Params=354M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 8,620 | 659 | 24 | |
| MHAHadSize=Base, Params=701M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 13,317 | 1,172 | 13,974 | |
| MHAHadSize=Medium, Params=1.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 24,551 | 1,982 | 8,266 | |
| MHAHadSize=Large, Params=2.4B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 32,336 | 2,883 | 5,683 | |
| MHAHadSize=XL, Params=3.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 48,030 | 4,280 | 3,827 | |
| MHAHadSize=XXL, Params=5.1B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03 | 60,584 | 5,551 | 2,951 |