List your company
ResearchFormerly WizWand

Training Optimization Efficiency on H200 Performance Sweep

38.8Peak Memory Usage (GiB)FORGE
35.7656.2876.897.32Jun 22, 2026
Updated 4mo ago

Evaluation results

MethodLinks
FORGEBatch Size (BS)=1, Sequence Length (SEQ)=10242026.0638.81.58
bnb 8-bitBatch Size (BS)=1, Sequence Length (SEQ)=10242026.0645.60.47
FORGEBatch Size (BS)=2, Sequence Length (SEQ)=10242026.0645.71.66
FORGEBatch Size (BS)=1, Sequence Length (SEQ)=20482026.0645.71.61
bnb 8-bitBatch Size (BS)=2, Sequence Length (SEQ)=10242026.0645.90.55
bnb 8-bitBatch Size (BS)=1, Sequence Length (SEQ)=20482026.0645.90.57
bnb 8-bitBatch Size (BS)=4, Sequence Length (SEQ)=10242026.0646.30.67
bnb 8-bitBatch Size (BS)=2, Sequence Length (SEQ)=20482026.0657.60.7
bnb 8-bitBatch Size (BS)=1, Sequence Length (SEQ)=40962026.0657.60.7
FORGEBatch Size (BS)=4, Sequence Length (SEQ)=10242026.0659.41.52
FORGEBatch Size (BS)=2, Sequence Length (SEQ)=20482026.0659.41.5
FORGEBatch Size (BS)=1, Sequence Length (SEQ)=40962026.0659.41.47
fused AdamWBatch Size (BS)=1, Sequence Length (SEQ)=10242026.0675.21
fused AdamWBatch Size (BS)=2, Sequence Length (SEQ)=10242026.0675.61
fused AdamWBatch Size (BS)=1, Sequence Length (SEQ)=20482026.0675.61
bnb 8-bitBatch Size (BS)=4, Sequence Length (SEQ)=20482026.0685.10.81
bnb 8-bitBatch Size (BS)=2, Sequence Length (SEQ)=40962026.0685.10.81
FORGEBatch Size (BS)=4, Sequence Length (SEQ)=20482026.0686.81.41
FORGEBatch Size (BS)=2, Sequence Length (SEQ)=40962026.0686.81.37
fused AdamWBatch Size (BS)=4, Sequence Length (SEQ)=10242026.0687.31
fused AdamWBatch Size (BS)=2, Sequence Length (SEQ)=20482026.0687.31
fused AdamWBatch Size (BS)=1, Sequence Length (SEQ)=40962026.0687.31
fused AdamWBatch Size (BS)=4, Sequence Length (SEQ)=20482026.06114.81
fused AdamWBatch Size (BS)=2, Sequence Length (SEQ)=40962026.06114.81