Training Efficiency Profiling on Qwen3 family S=1024
100.6Per-Step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEModel Size=Qwen3-0.6B2026.06 | 100.6 | 6.6 | |
| FORGEModel Size=Qwen3-1.7B2026.06 | 102.7 | 12.5 | |
| fused AdamWModel Size=Qwen3-0.6B2026.06 | 106.4 | 8.2 | |
| vanilla AdamWModel Size=Qwen3-0.6B2026.06 | 111.4 | 8.2 | |
| fused AdamWModel Size=Qwen3-1.7B2026.06 | 114.3 | 18.2 | |
| vanilla AdamWModel Size=Qwen3-1.7B2026.06 | 127.2 | 20.3 | |
| FORGEModel Size=Qwen3-4B2026.06 | 132.6 | 23.9 | |
| FORGEModel Size=Qwen3-8B2026.06 | 135.1 | 40.5 | |
| bnb 8-bit AdamWModel Size=Qwen3-0.6B2026.06 | 135.7 | 5.7 | |
| FlashOptimModel Size=Qwen3-0.6B2026.06 | 136.1 | 7.5 | |
| GaLore r=128Model Size=Qwen3-0.6B2026.06 | 137.6 | 5.4 | |
| GaLore r=128Model Size=Qwen3-1.7B2026.06 | 138.7 | 9.7 | |
| FlashOptimModel Size=Qwen3-1.7B2026.06 | 139.2 | 15.8 | |
| FORGEModel Size=Qwen3-14B2026.06 | 152.1 | 68.3 | |
| fused AdamWModel Size=Qwen3-4B2026.06 | 159.4 | 40.4 | |
| APOLLO-Mini r=256Model Size=Qwen3-0.6B2026.06 | 160.6 | 5.7 | |
| APOLLO-Mini r=256Model Size=Qwen3-1.7B2026.06 | 162.3 | 10.1 | |
| bnb 8-bit AdamWModel Size=Qwen3-1.7B2026.06 | 166.7 | 11.3 | |
| optimi.gradient_releaseModel Size=Qwen3-0.6B2026.06 | 174 | 12.4 | |
| optimi.gradient_releaseModel Size=Qwen3-1.7B2026.06 | 177.8 | 30 | |
| GaLore r=128Model Size=Qwen3-4B2026.06 | 180 | 19.4 | |
| FlashOptimModel Size=Qwen3-4B2026.06 | 181.4 | 33.5 | |
| fused AdamWModel Size=Qwen3-8B2026.06 | 187.1 | 76.8 | |
| vanilla AdamWModel Size=Qwen3-4B2026.06 | 188.6 | 46.5 | |
| FlashOptimModel Size=Qwen3-8B2026.06 | 196.9 | 62.1 | |
| GaLore r=128Model Size=Qwen3-8B2026.06 | 199.7 | 39.2 | |
| APOLLO-Mini r=256Model Size=Qwen3-4B2026.06 | 215.1 | 20.4 | |
| optimi.gradient_releaseModel Size=Qwen3-4B2026.06 | 230.9 | 66.9 | |
| APOLLO-Mini r=256Model Size=Qwen3-8B2026.06 | 243.7 | 40.3 | |
| vanilla AdamWModel Size=Qwen3-8B2026.06 | 246.1 | 92 | |
| optimi.gradient_releaseModel Size=Qwen3-8B2026.06 | 248.5 | 130.3 | |
| bnb 8-bit AdamWModel Size=Qwen3-4B2026.06 | 262.7 | 24.7 | |
| GaLore r=128Model Size=Qwen3-14B2026.06 | 271.7 | 66.2 | |
| FlashOptimModel Size=Qwen3-14B2026.06 | 300.3 | 108.6 | |
| APOLLO-Mini r=256Model Size=Qwen3-14B2026.06 | 346.3 | 67.8 | |
| bnb 8-bit AdamWModel Size=Qwen3-8B2026.06 | 375.3 | 46.5 | |
| AdaLomoModel Size=Qwen3-0.6B2026.06 | 517.5 | 4.6 | |
| AdaLomoModel Size=Qwen3-1.7B2026.06 | 520.3 | 7.6 | |
| bnb 8-bit AdamWModel Size=Qwen3-14B2026.06 | 600.8 | 83.7 | |
| AdaLomoModel Size=Qwen3-4B2026.06 | 801.1 | 14.2 | |
| AdaLomoModel Size=Qwen3-8B2026.06 | 919.4 | 28.2 | |
| AdaLomoModel Size=Qwen3-14B2026.06 | 1,230.7 | 44.9 |