Fine-tuning on Alpaca (train)
0.5622LossAdamW16
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AdamW16Model=Llama_3.2_3B, State (MB)=12256.3125 ± 0.0000, Time (s)=895.5152 ± 4.45522026.05 | 0.5622 | 1.4073 | |
| AdamW16Model=Llama_3.2_1B, State (MB)=4714.5000 ± 0.0000, Time (s)=1731.4853 ± 22.33832026.05 | 0.7207 | 1.5548 | |
| GaLore-AW32Model=Llama_3.2_3B, State (MB)=24511.3125 ± 0.0000, Time (s)=881.6705 ± 4.40142026.05 | 0.7621 | 1.6311 | |
| BAOC0.5Model=Llama_3.2_3B, State (MB)=6127.4092 ± 0.0000, Time (s)=824.0374 ± 4.12012026.05 | 0.7944 | 1.6526 | |
| GaLore-AW32Model=Llama_3.2_1B, State (MB)=9428.5000 ± 0.0000, Time (s)=1733.7957 ± 19.62312026.05 | 0.8111 | 1.6861 | |
| AdamW8Model=Llama_3.2_3B, State (MB)=6224.9395 ± 0.0000, Time (s)=876.4541 ± 4.42652026.05 | 0.815 | 1.6423 | |
| AdamW8Model=Llama_3.2_1B, State (MB)=2394.5488 ± 0.0000, Time (s)=1720.7466 ± 28.29412026.05 | 0.8721 | 1.7859 | |
| BAOC0.5Model=Llama_3.2_1B, State (MB)=2140.1338 ± 0.0000, Time (s)=1717.8212 ± 17.05482026.05 | 0.8739 | 1.7823 | |
| Adam-miniModel=Llama_3.2_1B, State (MB)=4715.5338 ± 0.0000, Time (s)=1729.2948 ± 30.95042026.05 | 1.1298 | 2.2214 | |
| GaLore-AFModel=Llama_3.2_3B, State (MB)=6.9551 ± 0.0000, Time (s)=890.5379 ± 4.25832026.05 | 1.1854 | 2.9719 | |
| Adam-miniModel=Llama_3.2_3B, State (MB)=12258.1823 ± 0.0000, Time (s)=869.8237 ± 4.34502026.05 | 1.207 | 2.2555 | |
| GaLore-AFModel=Llama_3.2_1B, State (MB)=3.4351 ± 0.0000, Time (s)=1768.7772 ± 17.25822026.05 | 1.368 | 3.7171 |