Language Modeling on PG19 T5 (val)
15.31PPLXPrimer
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PrimerParams=110M, Steps=920K, Baseline Compute=1M2021.09 | 15.31 | 1.81 | |
| Primer-EZParams=110M, Steps=896K, Baseline Compute=1M2021.09 | 15.37 | 1.98 | |
| Transformer++Params=110M, Steps=998K, Baseline Compute=1M2021.09 | 15.64 | 1.2 | |
| Vanilla TransformerParams=110M, Steps=1M, Baseline Compute=1M2021.09 | 15.83 | — | |
| Transformer+GELUParams=110M, Steps=998K, Baseline Compute=1M2021.09 | 15.84 | 0.95 | |
| Primer-EZParams=110M, Steps=471K, Baseline Compute=525K2021.09 | 15.84 | 1.74 | |
| PrimerParams=110M, Steps=483K, Baseline Compute=525K2021.09 | 15.96 | 1.68 | |
| Transformer++Params=110M, Steps=524K, Baseline Compute=525K2021.09 | 16.15 | 1.18 | |
| Transformer+GELUParams=110M, Steps=524K, Baseline Compute=525K2021.09 | 16.35 | 1.01 | |
| Vanilla TransformerParams=110M, Steps=525K, Baseline Compute=525K2021.09 | 16.39 | — |