LLM Training Step Performance on Qwen3-4B S=4096 (base variant)
214Per-Step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 214 | 43.2 | |
| fused AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 299.1 | 56.8 | |
| GaLore r=128†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 313.9 | 35.8 | |
| FlashOptimBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 331.3 | 67.8 | |
| vanilla AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 333.8 | 56.8 | |
| APOLLO-Mini r=256†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 358.3 | 36.8 | |
| bnb 8-bit AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 463.4 | 41.1 | |
| AdaLomo†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 925.5 | 33.5 |