Training Iteration Efficiency on Qwen3-8B base (train)
131.6Per-Step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 131.6 | 36.7 | |
| FlashOptimHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 178.8 | 43.5 | |
| fused AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 184.8 | 76.6 | |
| GaLore r=128Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 196.1 | 39 | |
| optimi.gradient_releaseHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 230 | 66.4 | |
| APOLLO-Mini r=256Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 240 | 40.1 | |
| vanilla AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 243.5 | 91.8 | |
| bnb 8-bit AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 370.8 | 46.3 | |
| AdaLomoHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 909.7 | 25.2 |