Training Efficiency Profiling on Qwen3 profiling S=2048 (train)
104.1Per-Step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEModel Size=Qwen3-0.6B2026.06 | 104.1 | 10 | |
| FORGEModel Size=Qwen3-1.7B2026.06 | 106.4 | 16.8 | |
| fused AdamWModel Size=Qwen3-0.6B2026.06 | 109.5 | 11.6 | |
| vanilla AdamWModel Size=Qwen3-0.6B2026.06 | 113.7 | 11.6 | |
| fused AdamWModel Size=Qwen3-1.7B2026.06 | 117.1 | 22 | |
| vanilla AdamWModel Size=Qwen3-1.7B2026.06 | 129.9 | 22 | |
| FORGEModel Size=Qwen3-4B2026.06 | 135.4 | 30.2 | |
| bnb 8-bit AdamWModel Size=Qwen3-0.6B2026.06 | 138.8 | 9.1 | |
| FlashOptimModel Size=Qwen3-0.6B2026.06 | 138.8 | 12 | |
| GaLore r=128Model Size=Qwen3-0.6B2026.06 | 140.2 | 8.8 | |
| GaLore r=128Model Size=Qwen3-1.7B2026.06 | 141.8 | 13.5 | |
| FlashOptimModel Size=Qwen3-1.7B2026.06 | 141.9 | 23.3 | |
| FORGEModel Size=Qwen3-8B2026.06 | 155 | 48.1 | |
| APOLLO-Mini r=256Model Size=Qwen3-0.6B2026.06 | 163.2 | 9.1 | |
| APOLLO-Mini r=256Model Size=Qwen3-1.7B2026.06 | 164.8 | 13.9 | |
| bnb 8-bit AdamWModel Size=Qwen3-1.7B2026.06 | 169.7 | 15.1 | |
| fused AdamWModel Size=Qwen3-4B2026.06 | 171 | 43.9 | |
| optimi.gradient_releaseModel Size=Qwen3-0.6B2026.06 | 176.6 | 20.3 | |
| optimi.gradient_releaseModel Size=Qwen3-1.7B2026.06 | 182.1 | 47.1 | |
| GaLore r=128Model Size=Qwen3-4B2026.06 | 182.8 | 22.8 | |
| FlashOptimModel Size=Qwen3-4B2026.06 | 184.5 | 47.3 | |
| vanilla AdamWModel Size=Qwen3-4B2026.06 | 199.4 | 46.9 | |
| APOLLO-Mini r=256Model Size=Qwen3-4B2026.06 | 224.6 | 23.8 | |
| optimi.gradient_releaseModel Size=Qwen3-4B2026.06 | 232.8 | 103.4 | |
| FORGEModel Size=Qwen3-14B2026.06 | 248.7 | 78.7 | |
| fused AdamWModel Size=Qwen3-8B2026.06 | 249.3 | 77.2 | |
| GaLore r=128Model Size=Qwen3-8B2026.06 | 251.6 | 39.7 | |
| FlashOptimModel Size=Qwen3-8B2026.06 | 259.1 | 84.9 | |
| bnb 8-bit AdamWModel Size=Qwen3-4B2026.06 | 275.3 | 28.2 | |
| APOLLO-Mini r=256Model Size=Qwen3-8B2026.06 | 297.7 | 40.7 | |
| vanilla AdamWModel Size=Qwen3-8B2026.06 | 303.8 | 92.5 | |
| GaLore r=128Model Size=Qwen3-14B2026.06 | 419.2 | 66.5 | |
| bnb 8-bit AdamWModel Size=Qwen3-8B2026.06 | 433.4 | 47 | |
| APOLLO-Mini r=256Model Size=Qwen3-14B2026.06 | 479.5 | 68.3 | |
| AdaLomoModel Size=Qwen3-0.6B2026.06 | 525 | 8 | |
| AdaLomoModel Size=Qwen3-1.7B2026.06 | 531.1 | 11.8 | |
| bnb 8-bit AdamWModel Size=Qwen3-14B2026.06 | 744.7 | 84.2 | |
| AdaLomoModel Size=Qwen3-4B2026.06 | 813.3 | 20.5 | |
| AdaLomoModel Size=Qwen3-8B2026.06 | 933.3 | 34 | |
| AdaLomoModel Size=Qwen3-14B2026.06 | 1,307.6 | 53.5 |