LLM Training Step Performance on Qwen3-8B S=512 base variant
140.8Per-Step Latency (ms)FORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 140.8 | 36.7 | |
| fused AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 203.3 | 76.6 | |
| FlashOptimBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 214.6 | 43.5 | |
| GaLore r=128†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 219.6 | 39 | |
| optimi.gradient_releaseBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 253 | 66.4 | |
| APOLLO-Mini r=256†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 271.2 | 40.1 | |
| bnb 8-bit AdamWBatch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 463.5 | 46.3 | |
| AdaLomo†Batch Size=1, Precision=BF16-everywhere, Flash Attention 3=true, Liger Kernel=true, Hardware=H100 SXM 80 GB, Model Variant=base variant2026.06 | 1,000.7 | 25.2 |