Language Modeling on WikiText-103 v1 (val)
2.29Training LossFOAM
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| FOAMf=75, τ=0.4, ϵ₀=10−12, ϵmax=1 × 10−102026.06 | 2.29 | 18.22 | 622 | |
| AdamWf=N/A, τ=N/A, ϵ₀=10−12, ϵmax=N/A2026.06 | 2.3 | 18.37 | 609 | |
| FOAMf=75, τ=0.4, ϵ₀=10−12, ϵmax=2 × 10−102026.06 | 2.3 | 18.22 | 621 | |
| FOAMf=75, τ=0.4, ϵ₀=10−12, ϵmax=3 × 10−102026.06 | 2.3 | 18.27 | 625 | |
| stale Shampoof=100, τ=N/A, ϵ₀=10−12, ϵmax=N/A2026.06 | 2.31 | 18.26 | 630 | |
| DR-Shampoof=75, τ=0.4, ϵ₀=10−12, ϵmax=N/A2026.06 | 2.31 | 18.33 | 623 | |
| SOAPf=75, τ=N/A, ϵ₀=10−12, ϵmax=N/A2026.06 | 2.31 | 18.33 | 642 |