Language Modeling on C4 T5 (val)
15.82PPLXSwitch Primer
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Switch PrimerParams=550M, Steps=900K, Baseline Compute=1M2021.09 | 15.82 | 1.56 | |
| Switch TransformerParams=550M, Steps=1M, Baseline Compute=1M2021.09 | 16.32 | — | |
| Switch PrimerParams=550M, Steps=474K, Baseline Compute=525K2021.09 | 16.56 | 1.45 | |
| Switch TransformerParams=550M, Steps=525K, Baseline Compute=525K2021.09 | 17.16 | — | |
| Synthesizer + Squared ReLUParams=145M, Steps=996K, Baseline Compute=1M2021.09 | 18.83 | 1.96 | |
| PrimerParams=110M, Steps=920K, Baseline Compute=1M2021.09 | 19.07 | 1.91 | |
| Primer-EZParams=110M, Steps=896K, Baseline Compute=1M2021.09 | 19.07 | 1.9 | |
| Transformer++Params=110M, Steps=998K, Baseline Compute=1M2021.09 | 19.28 | 1.64 | |
| Synthesizer + Squared ReLUParams=145M, Steps=523K, Baseline Compute=525K2021.09 | 19.55 | 1.74 | |
| SynthesizerParams=145M, Steps=1M, Baseline Compute=1M2021.09 | 19.57 | — | |
| Transformer+GELUParams=110M, Steps=998K, Baseline Compute=1M2021.09 | 19.58 | 1.26 | |
| Vanilla TransformerParams=110M, Steps=1M, Baseline Compute=1M2021.09 | 19.82 | — | |
| PrimerParams=110M, Steps=483K, Baseline Compute=525K2021.09 | 19.82 | 1.68 | |
| Primer-EZParams=110M, Steps=471K, Baseline Compute=525K2021.09 | 19.83 | 1.71 | |
| Evolved TransformerParams=110M, Steps=668K, Baseline Compute=1M2021.09 | 19.84 | 0.98 | |
| Transformer++Params=110M, Steps=524K, Baseline Compute=525K2021.09 | 20.03 | 1.52 | |
| Transformer+GELUParams=110M, Steps=524K, Baseline Compute=525K2021.09 | 20.34 | 1.2 | |
| SynthesizerParams=145M, Steps=525K, Baseline Compute=525K2021.09 | 20.35 | — | |
| Vanilla TransformerParams=110M, Steps=525K, Baseline Compute=525K2021.09 | 20.61 | — | |
| Evolved TransformerParams=110M, Steps=351K, Baseline Compute=525K2021.09 | 20.79 | 0.89 |