Training Iteration Efficiency on Qwen3-1.7B base (train)
101.7Per-Step Latency (ms)FORGE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FORGEHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 101.7 | 10.4 | |
| fused AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 114.5 | 18 | |
| vanilla AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 127.3 | 20.1 | |
| GaLore r=128Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 137.4 | 9.5 | |
| FlashOptimHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 138.1 | 10.4 | |
| APOLLO-Mini r=256Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 160.3 | 9.9 | |
| bnb 8-bit AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 164.8 | 11.1 | |
| optimi.gradient_releaseHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 177.2 | 15 | |
| AdaLomoHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 527.5 | 7 |