Training Efficiency Profiling on Qwen3 family S=512 (train profiling)
100.2Per-Step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEModel Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 100.2 | 4.9 | |
| FORGEModel Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 101.7 | 10.4 | |
| fused AdamWModel Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 107.3 | 6.7 | |
| vanilla AdamWModel Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 111.9 | 7.3 | |
| fused AdamWModel Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 114.5 | 18 | |
| vanilla AdamWModel Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 127.3 | 20.1 | |
| FORGEModel Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 129.8 | 20.6 | |
| FORGEModel Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 131.6 | 36.7 | |
| bnb 8-bit AdamWModel Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 134.6 | 4.2 | |
| FlashOptimModel Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 135.2 | 4.7 | |
| GaLore r=128Model Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 136.6 | 4 | |
| GaLore r=128Model Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 137.4 | 9.5 | |
| FlashOptimModel Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 138.1 | 10.4 | |
| FORGEModel Size=Qwen3-14B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 150.1 | 63.2 | |
| fused AdamWModel Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 158.4 | 40.1 | |
| APOLLO-Mini r=256Model Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 159.1 | 4.2 | |
| APOLLO-Mini r=256Model Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 160.3 | 9.9 | |
| bnb 8-bit AdamWModel Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 164.8 | 11.1 | |
| GaLore r=128Model Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 176.9 | 19.1 | |
| optimi.gradient_releaseModel Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 177.2 | 15 | |
| optimi.gradient_releaseModel Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 177.4 | 6.3 | |
| FlashOptimModel Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 178.8 | 43.5 | |
| FlashOptimModel Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 179 | 22.6 | |
| fused AdamWModel Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 184.8 | 76.6 | |
| vanilla AdamWModel Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 187.6 | 46.2 | |
| GaLore r=128Model Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 196.1 | 39 | |
| APOLLO-Mini r=256Model Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 212.4 | 20.1 | |
| optimi.gradient_releaseModel Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 225.6 | 33.5 | |
| optimi.gradient_releaseModel Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 230 | 66.4 | |
| APOLLO-Mini r=256Model Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 240 | 40.1 | |
| vanilla AdamWModel Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 243.5 | 91.8 | |
| GaLore r=128Model Size=Qwen3-14B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 247.2 | 65.9 | |
| FlashOptimModel Size=Qwen3-14B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 257.8 | 76.6 | |
| bnb 8-bit AdamWModel Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 260.6 | 24.4 | |
| APOLLO-Mini r=256Model Size=Qwen3-14B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 315 | 67.6 | |
| optimi.gradient_releaseModel Size=Qwen3-14B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 343.6 | 117.3 | |
| bnb 8-bit AdamWModel Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 370.8 | 46.3 | |
| AdaLomoModel Size=Qwen3-0.6B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 527.1 | 3.1 | |
| AdaLomoModel Size=Qwen3-1.7B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 527.5 | 7 | |
| bnb 8-bit AdamWModel Size=Qwen3-14B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 564.1 | 83.7 | |
| AdaLomoModel Size=Qwen3-4B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 809.4 | 12.1 | |
| AdaLomoModel Size=Qwen3-8B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 909.7 | 25.2 | |
| AdaLomoModel Size=Qwen3-14B, Batch Size=1, Precision=BF16, Hardware=H2002026.06 | 1,216.6 | 40.7 |