Training Iteration Efficiency on Qwen3-4B base (train)
129.8Per-step Latency (ms)FORGE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FORGEHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 129.8 | 20.6 | |
| fused AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 158.4 | 40.1 | |
| GaLore r=128Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 176.9 | 19.1 | |
| FlashOptimHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 179 | 22.6 | |
| vanilla AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 187.6 | 46.2 | |
| APOLLO-Mini r=256Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 212.4 | 20.1 | |
| optimi.gradient_releaseHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 225.6 | 33.5 | |
| bnb 8-bit AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 260.6 | 24.4 | |
| AdaLomoHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 809.4 | 12.1 |