Fine-tuning on GSM8K (test)
2.4887Test PerplexityGaLore-AF
Evaluation results
| Method | Links | |
|---|---|---|
| GaLore-AFModel=Llama_3.2_3B, State (MB)=6.9551 ± 0.0000, Time (s)=4191.0108 ± 162.96932026.05 | 2.4887 | |
| GaLore-AFModel=Llama_3.2_1B, State (MB)=3.4351 ± 0.0000, Time (s)=344.5018 ± 0.41292026.05 | 2.7346 | |
| Adam-miniModel=Llama_3.2_1B, State (MB)=4715.5338 ± 0.0000, Time (s)=338.5162 ± 0.79522026.05 | 3.0536 | |
| BAOC0.5Model=Llama_3.2_1B, State (MB)=2140.1338 ± 0.0000, Time (s)=336.2259 ± 0.27602026.05 | 3.403 | |
| GaLore-AW32Model=Llama_3.2_1B, State (MB)=9428.5000 ± 0.0000, Time (s)=339.3417 ± 1.72662026.05 | 3.4293 | |
| AdamW16Model=Llama_3.2_1B, State (MB)=4714.5000 ± 0.0000, Time (s)=339.7833 ± 1.30112026.05 | 3.4436 | |
| AdamW8Model=Llama_3.2_1B, State (MB)=2394.5488 ± 0.0000, Time (s)=335.7520 ± 1.35822026.05 | 3.4735 | |
| GaLore-AW32Model=Llama_3.2_3B, State (MB)=24511.3125 ± 0.0000, Time (s)=4125.9537 ± 165.64562026.05 | 3.5431 | |
| AdamW16Model=Llama_3.2_3B, State (MB)=12256.3125 ± 0.0000, Time (s)=4085.5169 ± 168.23402026.05 | 3.5495 | |
| BAOC0.5Model=Llama_3.2_3B, State (MB)=6127.4092 ± 0.0000, Time (s)=4030.8570 ± 157.74222026.05 | 3.5713 | |
| AdamW8Model=Llama_3.2_3B, State (MB)=6224.9395 ± 0.0000, Time (s)=3945.8413 ± 160.40612026.05 | 3.6559 | |
| Adam-miniModel=Llama_3.2_3B, State (MB)=12258.1823 ± 0.0000, Time (s)=4135.7124 ± 140.34102026.05 | 3.6833 |