Training Efficiency Measurement on Llama S=2048 on B200 3.1-8B (base variant)
178.7Per-step Latency (ms)FORGE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FORGEProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 178.7 | 47.6 | |
| fused AdamWProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 216.9 | 75.6 | |
| FlashOptimProtocol=Protocol B, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 223.1 | 82 | |
| GaLorer=128, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 244.4 | 37.7 | |
| vanilla AdamWProtocol=Protocol A, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 251.3 | 90.5 | |
| APOLLO-Minir=256, Protocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 282 | 38.9 | |
| bnb 8-bit AdamWProtocol=Protocol B, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 368.4 | 45.9 | |
| AdaLomoProtocol=Protocol B, Approximated AdamW Dynamics=true, Batch Size=1, Hardware=B200, Sequence Length=20482026.06 | 715.5 | 31.6 |