Training Optimization Efficiency on H200 Performance Sweep
38.8Peak Memory Usage (GiB)FORGE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FORGEBatch Size (BS)=1, Sequence Length (SEQ)=10242026.06 | 38.8 | 1.58 | |
| bnb 8-bitBatch Size (BS)=1, Sequence Length (SEQ)=10242026.06 | 45.6 | 0.47 | |
| FORGEBatch Size (BS)=2, Sequence Length (SEQ)=10242026.06 | 45.7 | 1.66 | |
| FORGEBatch Size (BS)=1, Sequence Length (SEQ)=20482026.06 | 45.7 | 1.61 | |
| bnb 8-bitBatch Size (BS)=2, Sequence Length (SEQ)=10242026.06 | 45.9 | 0.55 | |
| bnb 8-bitBatch Size (BS)=1, Sequence Length (SEQ)=20482026.06 | 45.9 | 0.57 | |
| bnb 8-bitBatch Size (BS)=4, Sequence Length (SEQ)=10242026.06 | 46.3 | 0.67 | |
| bnb 8-bitBatch Size (BS)=2, Sequence Length (SEQ)=20482026.06 | 57.6 | 0.7 | |
| bnb 8-bitBatch Size (BS)=1, Sequence Length (SEQ)=40962026.06 | 57.6 | 0.7 | |
| FORGEBatch Size (BS)=4, Sequence Length (SEQ)=10242026.06 | 59.4 | 1.52 | |
| FORGEBatch Size (BS)=2, Sequence Length (SEQ)=20482026.06 | 59.4 | 1.5 | |
| FORGEBatch Size (BS)=1, Sequence Length (SEQ)=40962026.06 | 59.4 | 1.47 | |
| fused AdamWBatch Size (BS)=1, Sequence Length (SEQ)=10242026.06 | 75.2 | 1 | |
| fused AdamWBatch Size (BS)=2, Sequence Length (SEQ)=10242026.06 | 75.6 | 1 | |
| fused AdamWBatch Size (BS)=1, Sequence Length (SEQ)=20482026.06 | 75.6 | 1 | |
| bnb 8-bitBatch Size (BS)=4, Sequence Length (SEQ)=20482026.06 | 85.1 | 0.81 | |
| bnb 8-bitBatch Size (BS)=2, Sequence Length (SEQ)=40962026.06 | 85.1 | 0.81 | |
| FORGEBatch Size (BS)=4, Sequence Length (SEQ)=20482026.06 | 86.8 | 1.41 | |
| FORGEBatch Size (BS)=2, Sequence Length (SEQ)=40962026.06 | 86.8 | 1.37 | |
| fused AdamWBatch Size (BS)=4, Sequence Length (SEQ)=10242026.06 | 87.3 | 1 | |
| fused AdamWBatch Size (BS)=2, Sequence Length (SEQ)=20482026.06 | 87.3 | 1 | |
| fused AdamWBatch Size (BS)=1, Sequence Length (SEQ)=40962026.06 | 87.3 | 1 | |
| fused AdamWBatch Size (BS)=4, Sequence Length (SEQ)=20482026.06 | 114.8 | 1 | |
| fused AdamWBatch Size (BS)=2, Sequence Length (SEQ)=40962026.06 | 114.8 | 1 |