Language Model Training on OpenMathInstruct-2
1.52Speedup RatioFORGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FORGEGranularity=per-tile, Full Adam dynamics=✓, Algebraically exact (fp32 state)=✓, Kernel-level fusion=✓, FP8 compatible=✓, Data-parallel / FSDP2=TP/SP, Backbone=Llama-3.1-8B, Hardware=H200, Batch size=1, Sequence length=512, Precision=bf162026.06 | 1.52 | 53 | |
| AdamWGranularity=per-step, Full Adam dynamics=✓, Algebraically exact (fp32 state)=✓, grad_W in HBM=✓, Data-parallel / FSDP2=✓, Backbone=Llama-3.1-8B, Hardware=H200, Batch size=1, Sequence length=512, Precision=bf162026.06 | 1 | — | |
| GaLoreGranularity=per-step, grad_W in HBM=✓, Data-parallel / FSDP2=✓, Backbone=Llama-3.1-8B, Hardware=H200, Batch size=1, Sequence length=512, Precision=bf162026.06 | 0.95 | 50 | |
| COATGranularity=per-step, Full Adam dynamics=✓, grad_W in HBM=✓, Kernel-level fusion=✓, FP8 compatible=✓, Data-parallel / FSDP2=✓, Backbone=Llama-3.1-8B, Hardware=H200, Batch size=1, Sequence length=512, Precision=bf162026.06 | 0.87 | — | |
| APOLLOGranularity=per-step, grad_W in HBM=✓, Data-parallel / FSDP2=✓, Backbone=Llama-3.1-8B, Hardware=H200, Batch size=1, Sequence length=512, Precision=bf162026.06 | 0.76 | 49 | |
| bnb 8-bitGranularity=per-param, Full Adam dynamics=✓, grad_W in HBM=✓, Data-parallel / FSDP2=✓, Backbone=Llama-3.1-8B, Hardware=H200, Batch size=1, Sequence length=512, Precision=bf162026.06 | 0.46 | 39 |