Training Efficiency Measurement on Llama 3.1-8B Base (S=512, B200)
3.1Peak Memory Usage (GiB)AdaLomo
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AdaLomoModel=Qwen3-0.6B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 3.1 | 584.9 | |
| GaLoreModel=Qwen3-0.6B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=1282026.06 | 3.9 | 229.7 | |
| bnb 8-bit AdamWModel=Qwen3-0.6B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 4.2 | 232.7 | |
| APOLLO-MiniModel=Qwen3-0.6B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=2562026.06 | 4.2 | 249.4 | |
| FlashOptimModel=Qwen3-0.6B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 4.6 | 221.5 | |
| FORGEModel=Qwen3-0.6B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 5.1 | 181.5 | |
| optimi.gradient_releaseModel=Qwen3-0.6B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 6.2 | 248.6 | |
| fused AdamWModel=Qwen3-0.6B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 6.7 | 186.9 | |
| AdaLomoModel=Qwen3-1.7B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 6.9 | 614.3 | |
| vanilla AdamWModel=Qwen3-0.6B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 7.2 | 188.4 | |
| GaLoreModel=Qwen3-1.7B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=1282026.06 | 9.4 | 267.3 | |
| APOLLO-MiniModel=Qwen3-1.7B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=2562026.06 | 9.8 | 246 | |
| FlashOptimModel=Qwen3-1.7B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 10.4 | 220.6 | |
| FORGEModel=Qwen3-1.7B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 10.7 | 180.6 | |
| bnb 8-bit AdamWModel=Qwen3-1.7B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 11 | 242.4 | |
| AdaLomoModel=Qwen3-4B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 12.1 | 905.6 | |
| optimi.gradient_releaseModel=Qwen3-1.7B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 15 | 255.9 | |
| fused AdamWModel=Qwen3-1.7B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 18 | 192.1 | |
| GaLoreModel=Qwen3-4B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=1282026.06 | 19.1 | 274.2 | |
| vanilla AdamWModel=Qwen3-1.7B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 20 | 150.5 | |
| APOLLO-MiniModel=Qwen3-4B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=2562026.06 | 20.1 | 347.2 | |
| FORGEModel=Qwen3-4B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 21.3 | 222.1 | |
| FlashOptimModel=Qwen3-4B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 22.6 | 311.3 | |
| AdaLomoProtocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 23.5 | 708 | |
| AdaLomoModel=Llama-3.1-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 23.5 | 708 | |
| bnb 8-bit AdamWModel=Qwen3-4B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 24.4 | 299.7 | |
| AdaLomoModel=Qwen3-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 25.2 | 946.7 | |
| optimi.gradient_releaseModel=Qwen3-4B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 33.5 | 293.3 | |
| FORGEProtocol=Protocol A, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 35.8 | 172.6 | |
| FORGEModel=Llama-3.1-8B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 35.8 | 172.6 | |
| GaLorer=128, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 37.2 | 242.1 | |
| GaLoreModel=Llama-3.1-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=1282026.06 | 37.2 | 242.1 | |
| FORGEModel=Qwen3-8B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 37.6 | 222 | |
| APOLLO-Minir=256, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 38.3 | 279.5 | |
| APOLLO-MiniModel=Llama-3.1-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=2562026.06 | 38.3 | 279.5 | |
| GaLoreModel=Qwen3-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=1282026.06 | 38.9 | 269.2 | |
| APOLLO-MiniModel=Qwen3-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=2562026.06 | 40 | 332.8 | |
| fused AdamWModel=Qwen3-4B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 40.1 | 237.4 | |
| FlashOptimProtocol=Protocol B, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 42.2 | 218.7 | |
| FlashOptimModel=Llama-3.1-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 42.2 | 218.7 | |
| FlashOptimModel=Qwen3-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 43.4 | 268.7 | |
| bnb 8-bit AdamWProtocol=Protocol B, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 45.3 | 364.6 | |
| bnb 8-bit AdamWModel=Llama-3.1-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 45.3 | 364.6 | |
| vanilla AdamWModel=Qwen3-4B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 46.2 | 251 | |
| bnb 8-bit AdamWModel=Qwen3-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 46.2 | 397.2 | |
| optimi.gradient_releaseProtocol=Protocol B, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 64.5 | 240.6 | |
| FORGEModel=Qwen3-14B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 64.5 | — | |
| optimi.gradient_releaseModel=Llama-3.1-8B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 64.5 | 240.6 | |
| GaLoreModel=Qwen3-14B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled, Rank=1282026.06 | 65.9 | 300.1 | |
| fused AdamWProtocol=Protocol A, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 75 | 205.2 | |
| fused AdamWModel=Llama-3.1-8B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 75 | 205.2 | |
| fused AdamWModel=Qwen3-8B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 76.5 | 252.6 | |
| FlashOptimModel=Qwen3-14B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 76.6 | 344.4 | |
| bnb 8-bit AdamWModel=Qwen3-14B, Protocol=B, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 83.6 | 555.3 | |
| vanilla AdamWProtocol=Protocol A, Batch Size=1, Attention=FlashAttention-4, Hardware=B200, Sequence Length=5122026.06 | 89.9 | 240.1 | |
| vanilla AdamWModel=Llama-3.1-8B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 89.9 | 240.1 | |
| vanilla AdamWModel=Qwen3-8B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 91.8 | 283.9 | |
| fused AdamWModel=Qwen3-14B, Protocol=A, Precision=BF16-everywhere, FlashAttention=FA4, Liger Kernel=Enabled2026.06 | 137.8 | 226.3 |