List your company
ResearchFormerly WizWand

Training Efficiency Profiling on Qwen3 (train)

3.1Peak GPU MemoryAdaLomo
-2.63236.05974.75113.441Jun 22, 2026
Updated 4mo ago

Evaluation results

MethodLinks
AdaLomoModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.063.1584.9
GaLore (r=128)Model Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.063.9229.7
bnb 8-bit AdamWModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.064.2232.7
APOLLO-Mini (r=256)Model Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.064.2249.4
FlashOptimModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.064.6221.5
FORGEModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=A, Batch Size=1, Precision=BF162026.065.1181.5
optimi.gradient_releaseModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=B, Batch Size=1, Precision=BF162026.066.2248.6
fused AdamWModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=A, Batch Size=1, Precision=BF162026.066.7186.9
FORGEModel Scale=Qwen3-0.6B, Sequence Length=1024, Protocol=A2026.067.1189.6
vanilla AdamWModel Scale=Qwen3-0.6B, Sequence Length=512, Protocol=A, Batch Size=1, Precision=BF162026.067.2188.4
FORGEModel Scale=Qwen3-1.7B, Sequence Length=512, Protocol=A2026.0610.7180.6
FORGEModel Scale=Qwen3-0.6B, Sequence Length=2048, Protocol=A2026.0611191.3
FORGEModel Scale=Qwen3-1.7B, Sequence Length=1024, Protocol=A2026.0613.2101.6
FORGEModel Scale=Qwen3-1.7B, Sequence Length=2048, Protocol=A2026.0618.3—
FORGEModel Scale=Qwen3-0.6B, Sequence Length=4096, Protocol=A2026.0618.9171.4
FORGEModel Scale=Qwen3-4B, Sequence Length=512, Protocol=A2026.0621.3222.1
FORGEModel Scale=Qwen3-4B, Sequence Length=1024, Protocol=A2026.0625.3229.3
FORGEModel Scale=Qwen3-1.7B, Sequence Length=4096, Protocol=A2026.0628.4100.4
FORGEModel Scale=Qwen3-4B, Sequence Length=2048, Protocol=A2026.0633231
FORGEModel Scale=Qwen3-8B, Sequence Length=512, Protocol=A2026.0637.6222
FORGEModel Scale=Qwen3-8B, Sequence Length=1024, Protocol=A2026.0642.3226
FORGEModel Scale=Qwen3-4B, Sequence Length=4096, Protocol=A2026.0648.8234.4
FORGEModel Scale=Qwen3-8B, Sequence Length=2048, Protocol=A2026.0651.8230.5
FORGEModel Scale=Qwen3-14B, Sequence Length=512, Protocol=A2026.0664.5—
FORGEModel Scale=Qwen3-8B, Sequence Length=4096, Protocol=A2026.0670.7256.3
FORGEModel Scale=Qwen3-14B, Sequence Length=1024, Protocol=A2026.0670.8246.1
GaLore (r=128)Model Scale=Qwen3-14B, Sequence Length=4096, Protocol=B2026.0676.4429.9
FORGEModel Scale=Qwen3-14B, Sequence Length=2048, Protocol=A2026.0683.7252.9
FORGEModel Scale=Qwen3-14B, Sequence Length=4096, Protocol=A2026.06109.4362.6
fused AdamWModel Scale=Qwen3-14B, Sequence Length=2048, Protocol=A2026.06138.5336
FlashOptimModel Scale=Qwen3-14B, Sequence Length=2048, Protocol=B2026.06146.4353.1