Efficiency Benchmarking on Llama 8B base variant on H200 3.1
99.1Per-step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGESequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Kernel Implementation=Triton-fronted2026.06 | 99.1 | 35.4 | |
| FORGESequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Kernel Implementation=Triton-fronted2026.06 | 101.5 | 38.8 | |
| FORGESequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Kernel Implementation=Triton-fronted2026.06 | 143 | 45.7 | |
| FlashOptimSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 148.5 | 42.3 | |
| fused AdamWSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 150.9 | 75 | |
| GaLore r=128Sequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 158.1 | 37.2 | |
| fused AdamWSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 159.4 | 75.2 | |
| GaLore r=128Sequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 160.8 | 37.4 | |
| FlashOptimSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 172.9 | 60.3 | |
| COAT-anchorSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 173.2 | 120.1 | |
| APOLLO-Mini r=256Sequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 199.6 | 38.4 | |
| APOLLO-Mini r=256Sequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 207.7 | 38.6 | |
| COAT-anchorSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 208.3 | 120.4 | |
| vanilla AdamWSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 209.2 | 90 | |
| vanilla AdamWSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 215.5 | 90.2 | |
| optimi.gradient_releaseSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 220.2 | 127.1 | |
| optimi.gradient_releaseSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 224 | 64.5 | |
| fused AdamWSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 230.8 | 75.6 | |
| GaLore r=128Sequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 235.5 | 37.8 | |
| FlashOptimSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 240.7 | 82.1 | |
| FORGESequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Kernel Implementation=Triton-fronted2026.06 | 273.4 | 59.4 | |
| APOLLO-Mini r=256Sequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 277.5 | 38.9 | |
| vanilla AdamWSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 284 | 90.5 | |
| COAT-anchorSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 301.8 | 121.6 | |
| bnb 8-bit AdamWSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 328.8 | 45.4 | |
| bnb 8-bit AdamWSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 339 | 45.6 | |
| COAT-actSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 364.1 | 120.1 | |
| COAT-actSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 366.5 | 121.1 | |
| COAT-actSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 367.9 | 120.4 | |
| fused AdamWSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 402.5 | 87.3 | |
| bnb 8-bit AdamWSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 408.4 | 45.9 | |
| GaLore r=128Sequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 411.9 | 47.5 | |
| FlashOptimSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 413.4 | 110.7 | |
| APOLLO-Mini r=256Sequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 435.3 | 48.6 | |
| COAT-optSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 445.8 | 76.7 | |
| COAT-actSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 448.4 | 122.6 | |
| vanilla AdamWSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 450.8 | 91.3 | |
| COAT-optSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 483.2 | 77 | |
| bnb 8-bit AdamWSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA32026.06 | 574.3 | 57.6 | |
| COAT-optSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 577.8 | 78.2 | |
| COAT-bothSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 636.5 | 76.7 | |
| COAT-bothSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 639.3 | 77.7 | |
| COAT-bothSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 639.4 | 77 | |
| AdaLomoSequence Length (S)=1024, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 643.8 | 26.3 | |
| AdaLomoSequence Length (S)=512, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 644.7 | 23.6 | |
| AdaLomoSequence Length (S)=2048, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 681.6 | 31.7 | |
| COAT-bothSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 721.9 | 79.1 | |
| COAT-optSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=FP32 master weights + autocast(BF16), Attention Mechanism=SDPA2026.06 | 781.9 | 96.1 | |
| AdaLomoSequence Length (S)=4096, Batch Size (BS)=1, Precision Regime=BF16-everywhere, Attention Mechanism=FA3, Optimization Type=approximated AdamW dynamics2026.06 | 802.6 | 42.5 |