Language Modeling on RoBERTa+cc100en (val)
9.84Validation PPLsMLP - deterministic
Evaluation Results
| Method | Links | |
|---|---|---|
| sMLP - deterministicModel Size (Parameters)=9.41B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=112, Speed (words per second)=144k2022.03 | 9.84 | |
| Switch Transformer - EnlargeModel Size (Parameters)=10.31B, Training Steps=100k, FLOPs=2.3T, Time to Quality (hours)=140, Speed (words per second)=126k2022.03 | 10.09 | |
| Switch TransformerModel Size (Parameters)=9.03B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=120, Speed (words per second)=139k2022.03 | 10.45 | |
| sMLP - deterministicModel Size (Parameters)=3.50B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=20.41, Speed (words per second)=192k2022.03 | 13.46 | |
| HASH LayersModel Size (Parameters)=3.51B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=21.08, Speed (words per second)=176k2022.03 | 13.57 | |
| GshardModel Size (Parameters)=9.03B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=198, Speed (words per second)=115k2022.03 | 14.5 | |
| Switch TransformerModel Size (Parameters)=3.48B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=20.67, Speed (words per second)=185k2022.03 | 15.31 | |
| STABLEMOESize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 16.22 | |
| BASE LayerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 16.36 | |
| Hash LayerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 16.37 | |
| Switch TransformerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 16.62 | |
| HASH LayersModel Size (Parameters)=10.29B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=100, Speed (words per second)=141k2022.03 | 16.69 | |
| Larger Transformer (deeper)Size=Base, # Shared Params=578M, # Expert Params=N/A, FLOPs=610B2022.04 | 17.93 | |
| Base LayersModel Size (Parameters)=3.60B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=21.13, Speed (words per second)=178k2022.03 | 18.16 | |
| Larger Transformer (wider)Size=Base, # Shared Params=578M, # Expert Params=N/A, FLOPs=610B2022.04 | 18.31 | |
| Standard TransformerSize=Large, # Shared Params=355M, # Expert Params=N/A, FLOPs=414B2022.04 | 18.86 | |
| GshardModel Size (Parameters)=3.48B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=27.54, Speed (words per second)=136k2022.03 | 19.26 | |
| STABLEMOESize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 19.28 | |
| Hash LayerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 19.63 | |
| Switch TransformerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 19.79 | |
| BASE LayerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 20.04 | |
| Base LayersModel Size (Parameters)=10.31B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=115.13, Speed (words per second)=135k, note=divergence during training2022.03 | 20.36 | |
| Standard TransformerSize=Base, # Shared Params=124M, # Expert Params=N/A, FLOPs=146B2022.04 | 23.02 | |
| Hash LayerConfiguration=layers=11, modules=1x64, Params=794M2021.06 | 26.99 | |
| Switch TransformerConfiguration=layers=11, modules=1x64, load_bal=0.1, Params=795M2021.06 | 27.41 | |
| Baseline TransformerConfiguration=layers=11, d=1024, D=4096, Params=266M2021.06 | 28.85 |