Training Iteration Efficiency on Llama-3.1-8B base (train)
99.1Latency (ms)FORGE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FORGEHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 99.1 | 35.4 | |
| FlashOptimHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 148.5 | 42.3 | |
| fused AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 150.9 | 75 | |
| GaLore r=128Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 158.1 | 37.2 | |
| APOLLO-Mini r=256Hardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 199.6 | 38.4 | |
| vanilla AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 209.2 | 90 | |
| optimi.gradient_releaseHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 224 | 64.5 | |
| bnb 8-bit AdamWHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 328.8 | 45.4 | |
| AdaLomoHardware=H200, BS=1, S=512, Precision=BF16, Attention=FA3, Kernel=Liger2026.06 | 644.7 | 23.6 |