Training Iteration Efficiency on Qwen3-0.6B base (train)
100.2Per-step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 100.2 | 4.9 | |
| FORGEBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 107.2 | 4.9 | |
| fused AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 107.3 | 6.7 | |
| vanilla AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 111.9 | 7.3 | |
| fused AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 118 | 6.7 | |
| vanilla AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 125.3 | 7.3 | |
| bnb 8-bit AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 134.6 | 4.2 | |
| FlashOptimHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 135.2 | 4.7 | |
| GaLore r=128Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 136.6 | 4 | |
| GaLore r=128†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 153 | 4 | |
| APOLLO-Mini r=256Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 159.1 | 4.2 | |
| FlashOptimBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 162.2 | 4.7 | |
| bnb 8-bit AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 168.3 | 4.2 | |
| optimi.gradient_releaseHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 177.4 | 6.3 | |
| APOLLO-Mini r=256†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 179.8 | 4.2 | |
| optimi.gradient_releaseBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 195.1 | 6.3 | |
| AdaLomoHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 527.1 | 3.1 | |
| AdaLomo†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 579.8 | 3.1 |