List your company
ResearchFormerly WizWand

Language Modeling on C4 T5 (val)

15.82PPLXSwitch Primer
15.621216.963118.30519.6469Sep 17, 2021
Updated 8mo ago

Evaluation results

MethodLinks
Switch PrimerParams=550M, Steps=900K, Baseline Compute=1M2021.0915.821.56
Switch TransformerParams=550M, Steps=1M, Baseline Compute=1M2021.0916.32—
Switch PrimerParams=550M, Steps=474K, Baseline Compute=525K2021.0916.561.45
Switch TransformerParams=550M, Steps=525K, Baseline Compute=525K2021.0917.16—
Synthesizer + Squared ReLUParams=145M, Steps=996K, Baseline Compute=1M2021.0918.831.96
PrimerParams=110M, Steps=920K, Baseline Compute=1M2021.0919.071.91
Primer-EZParams=110M, Steps=896K, Baseline Compute=1M2021.0919.071.9
Transformer++Params=110M, Steps=998K, Baseline Compute=1M2021.0919.281.64
Synthesizer + Squared ReLUParams=145M, Steps=523K, Baseline Compute=525K2021.0919.551.74
SynthesizerParams=145M, Steps=1M, Baseline Compute=1M2021.0919.57—
Transformer+GELUParams=110M, Steps=998K, Baseline Compute=1M2021.0919.581.26
Vanilla TransformerParams=110M, Steps=1M, Baseline Compute=1M2021.0919.82—
PrimerParams=110M, Steps=483K, Baseline Compute=525K2021.0919.821.68
Primer-EZParams=110M, Steps=471K, Baseline Compute=525K2021.0919.831.71
Evolved TransformerParams=110M, Steps=668K, Baseline Compute=1M2021.0919.840.98
Transformer++Params=110M, Steps=524K, Baseline Compute=525K2021.0920.031.52
Transformer+GELUParams=110M, Steps=524K, Baseline Compute=525K2021.0920.341.2
SynthesizerParams=145M, Steps=525K, Baseline Compute=525K2021.0920.35—
Vanilla TransformerParams=110M, Steps=525K, Baseline Compute=525K2021.0920.61—
Evolved TransformerParams=110M, Steps=351K, Baseline Compute=525K2021.0920.790.89