Training Efficiency Profiling on Qwen3 profiling S=4096 (train)
108Per-step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEModel Size=Qwen3-0.6B2026.06 | 108 | 16.8 | |
| FORGEModel Size=Qwen3-1.7B2026.06 | 112.6 | 25.3 | |
| fused AdamWModel Size=Qwen3-0.6B2026.06 | 113.6 | 18.4 | |
| vanilla AdamWModel Size=Qwen3-0.6B2026.06 | 118.1 | 18.4 | |
| fused AdamWModel Size=Qwen3-1.7B2026.06 | 139.4 | 30.6 | |
| bnb 8-bit AdamWModel Size=Qwen3-0.6B2026.06 | 142.8 | 15.9 | |
| FlashOptimModel Size=Qwen3-0.6B2026.06 | 142.9 | 19.9 | |
| GaLore r=128Model Size=Qwen3-0.6B2026.06 | 144.3 | 15.6 | |
| FlashOptimModel Size=Qwen3-1.7B2026.06 | 146.2 | 35 | |
| vanilla AdamWModel Size=Qwen3-1.7B2026.06 | 151.7 | 30.6 | |
| GaLore r=128Model Size=Qwen3-1.7B2026.06 | 156.7 | 22 | |
| APOLLO-Mini r=256Model Size=Qwen3-0.6B2026.06 | 167.4 | 15.9 | |
| optimi.gradient_releaseModel Size=Qwen3-0.6B2026.06 | 181.4 | 31.5 | |
| APOLLO-Mini r=256Model Size=Qwen3-1.7B2026.06 | 184.7 | 22.5 | |
| optimi.gradient_releaseModel Size=Qwen3-1.7B2026.06 | 186.2 | 68.5 | |
| bnb 8-bit AdamWModel Size=Qwen3-1.7B2026.06 | 191.5 | 23.7 | |
| FORGEModel Size=Qwen3-4B2026.06 | 200 | 43.2 | |
| fused AdamWModel Size=Qwen3-4B2026.06 | 271.9 | 56.8 | |
| FlashOptimModel Size=Qwen3-4B2026.06 | 276.1 | 67.8 | |
| GaLore r=128Model Size=Qwen3-4B2026.06 | 280.3 | 35.8 | |
| FORGEModel Size=Qwen3-8B2026.06 | 288.7 | 63.3 | |
| vanilla AdamWModel Size=Qwen3-4B2026.06 | 298 | 56.8 | |
| APOLLO-Mini r=256Model Size=Qwen3-4B2026.06 | 317.1 | 36.8 | |
| bnb 8-bit AdamWModel Size=Qwen3-4B2026.06 | 370.7 | 41.1 | |
| fused AdamWModel Size=Qwen3-8B2026.06 | 429 | 91.5 | |
| GaLore r=128Model Size=Qwen3-8B2026.06 | 429.2 | 51.5 | |
| FlashOptimModel Size=Qwen3-8B2026.06 | 438.5 | 115.4 | |
| APOLLO-Mini r=256Model Size=Qwen3-8B2026.06 | 460.3 | 52.6 | |
| vanilla AdamWModel Size=Qwen3-8B2026.06 | 479 | 93.3 | |
| FORGEModel Size=Qwen3-14B2026.06 | 480.7 | 99.4 | |
| AdaLomoModel Size=Qwen3-0.6B2026.06 | 534.1 | 14.8 | |
| AdaLomoModel Size=Qwen3-1.7B2026.06 | 537.3 | 20.4 | |
| bnb 8-bit AdamWModel Size=Qwen3-8B2026.06 | 605.8 | 61.3 | |
| GaLore r=128Model Size=Qwen3-14B2026.06 | 737 | 76.5 | |
| APOLLO-Mini r=256Model Size=Qwen3-14B2026.06 | 782.3 | 78.2 | |
| AdaLomoModel Size=Qwen3-4B2026.06 | 841.4 | 33.5 | |
| AdaLomoModel Size=Qwen3-8B2026.06 | 1,032.1 | 45.8 | |
| bnb 8-bit AdamWModel Size=Qwen3-14B2026.06 | 1,050.3 | 97.1 | |
| AdaLomoModel Size=Qwen3-14B2026.06 | 1,540.8 | 70.7 |