Training Efficiency Profiling on Qwen3 (train)
3.1Peak GPU MemoryAdaLomo
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AdaLomoModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.06 | 3.1 | 584.9 | |
| GaLore (r=128)Model Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.06 | 3.9 | 229.7 | |
| bnb 8-bit AdamWModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.06 | 4.2 | 232.7 | |
| APOLLO-Mini (r=256)Model Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.06 | 4.2 | 249.4 | |
| FlashOptimModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.06 | 4.6 | 221.5 | |
| FORGEModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=A, Batch Size=1, Precision=BF162026.06 | 5.1 | 181.5 | |
| optimi.gradient_releaseModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.06 | 6.2 | 248.6 | |
| fused AdamWModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=A, Batch Size=1, Precision=BF162026.06 | 6.7 | 186.9 | |
| FORGEModel Scale=Qwen3-0.6B, Sequence Length=1024, Protocol=A2026.06 | 7.1 | 189.6 | |
| vanilla AdamWModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=A, Batch Size=1, Precision=BF162026.06 | 7.2 | 188.4 | |
| FORGEModel Scale=Qwen3-1.7B, Sequence Length=512, Protocol=A2026.06 | 10.7 | 180.6 | |
| FORGEModel Scale=Qwen3-0.6B, Sequence Length=2048, Protocol=A2026.06 | 11 | 191.3 | |
| FORGEModel Scale=Qwen3-1.7B, Sequence Length=1024, Protocol=A2026.06 | 13.2 | 101.6 | |
| FORGEModel Scale=Qwen3-1.7B, Sequence Length=2048, Protocol=A2026.06 | 18.3 | — | |
| FORGEModel Scale=Qwen3-0.6B, Sequence Length=4096, Protocol=A2026.06 | 18.9 | 171.4 | |
| FORGEModel Scale=Qwen3-4B, Sequence Length=512, Protocol=A2026.06 | 21.3 | 222.1 | |
| FORGEModel Scale=Qwen3-4B, Sequence Length=1024, Protocol=A2026.06 | 25.3 | 229.3 | |
| FORGEModel Scale=Qwen3-1.7B, Sequence Length=4096, Protocol=A2026.06 | 28.4 | 100.4 | |
| FORGEModel Scale=Qwen3-4B, Sequence Length=2048, Protocol=A2026.06 | 33 | 231 | |
| FORGEModel Scale=Qwen3-8B, Sequence Length=512, Protocol=A2026.06 | 37.6 | 222 | |
| FORGEModel Scale=Qwen3-8B, Sequence Length=1024, Protocol=A2026.06 | 42.3 | 226 | |
| FORGEModel Scale=Qwen3-4B, Sequence Length=4096, Protocol=A2026.06 | 48.8 | 234.4 | |
| FORGEModel Scale=Qwen3-8B, Sequence Length=2048, Protocol=A2026.06 | 51.8 | 230.5 | |
| FORGEModel Scale=Qwen3-14B, Sequence Length=512, Protocol=A2026.06 | 64.5 | — | |
| FORGEModel Scale=Qwen3-8B, Sequence Length=4096, Protocol=A2026.06 | 70.7 | 256.3 | |
| FORGEModel Scale=Qwen3-14B, Sequence Length=1024, Protocol=A2026.06 | 70.8 | 246.1 | |
| GaLore (r=128)Model Scale=Qwen3-14B, Sequence Length=4096, Protocol=B2026.06 | 76.4 | 429.9 | |
| FORGEModel Scale=Qwen3-14B, Sequence Length=2048, Protocol=A2026.06 | 83.7 | 252.9 | |
| FORGEModel Scale=Qwen3-14B, Sequence Length=4096, Protocol=A2026.06 | 109.4 | 362.6 | |
| fused AdamWModel Scale=Qwen3-14B, Sequence Length=2048, Protocol=A2026.06 | 138.5 | 336 | |
| FlashOptimModel Scale=Qwen3-14B, Sequence Length=2048, Protocol=B2026.06 | 146.4 | 353.1 |