Fine-tuning on GSM8K (train)
0.3347Training LossAdamW16
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AdamW16Model=Llama_3.2_3B, State (MB)=12256.3125 ± 0.0000, Time (s)=4085.5169 ± 168.23402026.05 | 0.3347 | 1.2031 | |
| GaLore-AW32Model=Llama_3.2_3B, State (MB)=24511.3125 ± 0.0000, Time (s)=4125.9537 ± 165.64562026.05 | 0.3804 | 1.2343 | |
| AdamW8Model=Llama_3.2_3B, State (MB)=6224.9395 ± 0.0000, Time (s)=3945.8413 ± 160.40612026.05 | 0.3886 | 1.2522 | |
| BAOC0.5Model=Llama_3.2_3B, State (MB)=6127.4092 ± 0.0000, Time (s)=4030.8570 ± 157.74222026.05 | 0.3904 | 1.2501 | |
| AdamW16Model=Llama_3.2_1B, State (MB)=4714.5000 ± 0.0000, Time (s)=339.7833 ± 1.30112026.05 | 0.4636 | 1.3168 | |
| GaLore-AW32Model=Llama_3.2_1B, State (MB)=9428.5000 ± 0.0000, Time (s)=339.3417 ± 1.72662026.05 | 0.4838 | 1.3357 | |
| BAOC0.5Model=Llama_3.2_1B, State (MB)=2140.1338 ± 0.0000, Time (s)=336.2259 ± 0.27602026.05 | 0.4876 | 1.3444 | |
| AdamW8Model=Llama_3.2_1B, State (MB)=2394.5488 ± 0.0000, Time (s)=335.7520 ± 1.35822026.05 | 0.4895 | 1.3419 | |
| Adam-miniModel=Llama_3.2_3B, State (MB)=12258.1823 ± 0.0000, Time (s)=4135.7124 ± 140.34102026.05 | 0.5216 | 1.3837 | |
| Adam-miniModel=Llama_3.2_1B, State (MB)=4715.5338 ± 0.0000, Time (s)=338.5162 ± 0.79522026.05 | 0.7468 | 1.695 | |
| GaLore-AFModel=Llama_3.2_3B, State (MB)=6.9551 ± 0.0000, Time (s)=4191.0108 ± 162.96932026.05 | 0.8065 | 2.1099 | |
| GaLore-AFModel=Llama_3.2_1B, State (MB)=3.4351 ± 0.0000, Time (s)=344.5018 ± 0.41292026.05 | 0.9615 | 2.5308 |