List your company
ResearchFormerly WizWand

Decode-phase Efficiency Benchmarking on H100-80GB GPU

13Peak Memory (MB)Baseline
-2,409.8413,944.3330,298.546,652.67Mar 9, 2026
Updated 7mo ago

Evaluation results

MethodLinks
BaselineSize=Base, Params=757M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03131,22113
BaselineSize=Medium, Params=1.7B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03252,0757,896
BaselineSize=Large, Params=2.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03333,0615,353
BaselineSize=XL, Params=3.9B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03494,5443,605
BaselineSize=XXL, Params=5.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.03625,9202,767
MHAHadSize=Tiny, Params=117M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.033,45427559,478
BaselineSize=Tiny, Params=124M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.033,49027260
MHAHadSize=Small, Params=329M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.038,52563525,797
BaselineSize=Small, Params=354M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.038,62065924
MHAHadSize=Base, Params=701M, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.0313,3171,17213,974
MHAHadSize=Medium, Params=1.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.0324,5511,9828,266
MHAHadSize=Large, Params=2.4B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.0332,3362,8835,683
MHAHadSize=XL, Params=3.6B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.0348,0304,2803,827
MHAHadSize=XXL, Params=5.1B, Batch size=1024, Sequence length=2, Precision=BF16, Hardware=H100-80GB GPU2026.0360,5845,5512,951