Language Modeling on OpenWebText GPT-2 (test)
17.94PerplexityAdamCPR (Kappa-IP)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AdamCPR (Kappa-IP)Number of random seeds=3, Model=GPT2s2023.11 | 17.94 | — | |
| AdamCPR (Kappa-WS)Warm start steps (s)=5k (1x), Number of random seeds=3, Model=GPT2s2023.11 | 18.02 | 0.0258 | |
| AdamCPR (Kappa-WS)Warm start steps (s)=10k (2x), Number of random seeds=3, Model=GPT2s2023.11 | 18.03 | 0.0178 | |
| AdamWWeight decay coefficient (gamma)=1e-2, Number of random seeds=3, Model=GPT2s2023.11 | 18.23 | 0.0113 | |
| AdamCPR (Kappa-WS)Warm start steps (s)=20k (4x), Number of random seeds=3, Model=GPT2s2023.11 | 18.24 | 0.032 | |
| Adam AdadecayWeight decay coefficient (gamma)=1e-2, Number of random seeds=1, Model=GPT2s2023.11 | 18.24 | — | |
| Adam AdadecayWeight decay coefficient (gamma)=1e-3, Number of random seeds=1, Model=GPT2s2023.11 | 18.42 | — | |
| Adam AWDWeight decay coefficient (gamma)=1e-3, Number of random seeds=1, Model=GPT2s2023.11 | 18.42 | — | |
| AdamWWeight decay coefficient (gamma)=1e-3, Number of random seeds=3, Model=GPT2s2023.11 | 18.45 | 0.0039 | |
| Adam AWDWeight decay coefficient (gamma)=1e-2, Number of random seeds=1, Model=GPT2s2023.11 | 18.47 | — | |
| Adam AWDWeight decay coefficient (gamma)=1e-1, Number of random seeds=1, Model=GPT2s2023.11 | 18.49 | — | |
| AdamWWeight decay coefficient (gamma)=1e-1, Number of random seeds=3, Model=GPT2s2023.11 | 18.86 | 0.0169 | |
| Adam AdadecayWeight decay coefficient (gamma)=1e-1, Number of random seeds=1, Model=GPT2s2023.11 | 18.87 | — |