Fine-tuning on Alpaca (test)
7.3804Test PerplexityAdam-mini
Evaluation results
| Method | Links | |
|---|---|---|
| Adam-miniModel=Llama_3.2_1B, State (MB)=4715.5338 ± 0.0000, Time (s)=1729.2948 ± 30.95042026.05 | 7.3804 | |
| AdamW16Model=Llama_3.2_1B, State (MB)=4714.5000 ± 0.0000, Time (s)=1731.4853 ± 22.33832026.05 | 7.7593 | |
| BAOC0.5Model=Llama_3.2_1B, State (MB)=2140.1338 ± 0.0000, Time (s)=1717.8212 ± 17.05482026.05 | 7.8588 | |
| GaLore-AW32Model=Llama_3.2_1B, State (MB)=9428.5000 ± 0.0000, Time (s)=1733.7957 ± 19.62312026.05 | 7.9558 | |
| AdamW16Model=Llama_3.2_3B, State (MB)=12256.3125 ± 0.0000, Time (s)=895.5152 ± 4.45522026.05 | 8.1776 | |
| AdamW8Model=Llama_3.2_1B, State (MB)=2394.5488 ± 0.0000, Time (s)=1720.7466 ± 28.29412026.05 | 8.2353 | |
| GaLore-AFModel=Llama_3.2_1B, State (MB)=3.4351 ± 0.0000, Time (s)=1768.7772 ± 17.25822026.05 | 8.3177 | |
| GaLore-AFModel=Llama_3.2_3B, State (MB)=6.9551 ± 0.0000, Time (s)=890.5379 ± 4.25832026.05 | 8.9715 | |
| GaLore-AW32Model=Llama_3.2_3B, State (MB)=24511.3125 ± 0.0000, Time (s)=881.6705 ± 4.40142026.05 | 9.1036 | |
| BAOC0.5Model=Llama_3.2_3B, State (MB)=6127.4092 ± 0.0000, Time (s)=824.0374 ± 4.12012026.05 | 9.5502 | |
| AdamW8Model=Llama_3.2_3B, State (MB)=6224.9395 ± 0.0000, Time (s)=876.4541 ± 4.42652026.05 | 9.6611 | |
| Adam-miniModel=Llama_3.2_3B, State (MB)=12258.1823 ± 0.0000, Time (s)=869.8237 ± 4.34502026.05 | 9.9557 |