LLM Training Step Performance on Qwen3-1.7B S=2048 (base variant)
113.8Per-Step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 113.8 | 16.8 | |
| fused AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 128.8 | 22 | |
| vanilla AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 145.5 | 22 | |
| GaLore r=128†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 158.8 | 13.5 | |
| FlashOptimBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 170.3 | 23.3 | |
| APOLLO-Mini r=256†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 186.2 | 13.9 | |
| optimi.gradient_releaseBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 200.3 | 47.1 | |
| bnb 8-bit AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 212.1 | 15.1 | |
| AdaLomo†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 584.2 | 11.8 |