Loss curve fitting across model sizes on MoE models (various sizes)
0.341ASMT MAPEASMT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ASMTLRmax=1e-4, Batch Size=5122025.12 | 0.341 | — | — | |
| ASMTLRmax=2e-3, Batch Size=5122025.12 | 0.574 | — | — | |
| ASMTLRmax=2e-4, Batch Size=5122025.12 | 0.957 | — | — | |
| CMMT (λ = 0.99)LRmax=2e-4, Batch Size=5122025.12 | — | — | 0.75 | |
| CMMT (λ = 0.99)LRmax=2e-3, Batch Size=5122025.12 | — | — | 0.525 | |
| CMMT (λ = 0.99)LRmax=1e-4, Batch Size=5122025.12 | — | — | 0.356 | |
| CMMT (λ = 0.999)LRmax=2e-4, Batch Size=5122025.12 | — | 1.039 | — | |
| CMMT (λ = 0.999)LRmax=2e-3, Batch Size=5122025.12 | — | 0.487 | — | |
| CMMT (λ = 0.999)LRmax=1e-4, Batch Size=5122025.12 | — | 0.373 | — |