Language Model Pre-training on FineWeb-Edu
14.13PerplexityMuown
Evaluation Results
| Method | Links | |
|---|---|---|
| Muownη (learning rate)=4 × 10⁻³, λ (weight decay)=0.0, Model size=500M, Seeds=32026.05 | 14.13 | |
| Muownη (learning rate)=2 × 10⁻³, λ (weight decay)=0.0, Model size=500M, Seeds=32026.05 | 14.14 | |
| Muownη (learning rate)=1 × 10⁻³, λ (weight decay)=0.0, Model size=500M, Seeds=32026.05 | 14.24 | |
| Muonη (learning rate)=2 × 10⁻³, λ (weight decay)=0.1, Model size=500M, Seeds=32026.05 | 14.37 | |
| Muonη (learning rate)=2 × 10⁻³, λ (weight decay)=0.0, Model size=500M, Seeds=32026.05 | 14.4 | |
| Muonη (learning rate)=1 × 10⁻³, λ (weight decay)=0.1, Model size=500M, Seeds=32026.05 | 14.44 | |
| SOAPη (learning rate)=2 × 10⁻³, λ (weight decay)=10⁻⁴, Model size=500M, Seeds=32026.05 | 14.47 | |
| SOAPη (learning rate)=1 × 10⁻³, λ (weight decay)=10⁻⁴, Model size=500M, Seeds=32026.05 | 14.49 | |
| SOAPη (learning rate)=4 × 10⁻³, λ (weight decay)=10⁻⁴, Model size=500M, Seeds=32026.05 | 14.53 | |
| Muonη (learning rate)=1 × 10⁻³, λ (weight decay)=0.0, Model size=500M, Seeds=32026.05 | 14.54 | |
| Muonη (learning rate)=4 × 10⁻³, λ (weight decay)=0.1, Model size=500M, Seeds=32026.05 | 28.37 | |
| Muonη (learning rate)=4 × 10⁻³, λ (weight decay)=0.0, Model size=500M, Seeds=32026.05 | 88.15 |