Training Efficiency Measurement on Llama base variant (S=1024) on B200 3.1-8B
178.1Latency (ms)FORGE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FORGEProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 178.1 | 39.8 | |
| fused AdamWProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 212.9 | 75.2 | |
| FlashOptimProtocol=Protocol B, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 218.1 | 60.3 | |
| optimi.gradient_releaseProtocol=Protocol B, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 239.1 | 127 | |
| vanilla AdamWProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 242.6 | 90.1 | |
| GaLorer=128, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 249.6 | 37.4 | |
| APOLLO-Minir=256, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 280.2 | 38.5 | |
| bnb 8-bit AdamWProtocol=Protocol B, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 368.6 | 45.5 | |
| AdaLomoProtocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=10242026.06 | 715.2 | 26.2 |