Training Efficiency Measurement on Llama 3.1-8B Base (S=4096, B200)
202Per-Step Latency (ms)FORGE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FORGEProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 202 | 63.4 | |
| FlashOptimProtocol=Protocol B, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 263.2 | 110.7 | |
| GaLorer=128, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 271.1 | 47.4 | |
| fused AdamWProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 285 | 91.3 | |
| APOLLO-Minir=256, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 313.5 | 48.5 | |
| vanilla AdamWProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 324.7 | 91.3 | |
| bnb 8-bit AdamWProtocol=Protocol B, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 406.9 | 57.6 | |
| AdaLomoProtocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=40962026.06 | 763.3 | 42.4 |