LLM Pretraining on C4 (Final val)
18.288PerplexitySoftMuon
Evaluation results
| Method | Links | |
|---|---|---|
| SoftMuonModel=LLaMa 130M, Batch size=512, Sequence length=256, Learning rate scheduling=cosine-annealing, Number of runs=32026.05 | 18.288 | |
| SoftSignumModel=LLaMa 130M, Batch size=512, Sequence length=256, Learning rate scheduling=cosine-annealing, Number of runs=32026.05 | 18.519 | |
| MuonModel=LLaMa 130M, Batch size=512, Sequence length=256, Learning rate scheduling=cosine-annealing, Number of runs=32026.05 | 18.591 | |
| SignumModel=LLaMa 130M, Batch size=512, Sequence length=256, Learning rate scheduling=cosine-annealing, Number of runs=32026.05 | 18.688 | |
| AdamWModel=LLaMa 130M, Batch size=512, Sequence length=256, Learning rate scheduling=cosine-annealing, Number of runs=32026.05 | 18.709 |