List your company
ResearchFormerly WizWand

Language Modeling on RoBERTa+cc100en (val)

9.84Validation PPLsMLP - deterministic
9.079614.212319.34524.4777Jun 8, 2021Jul 30, 2021Sep 20, 2021Nov 12, 2021Jan 3, 2022Feb 24, 2022Apr 18, 2022
Updated 8mo ago

Evaluation results

MethodLinks
sMLP - deterministicModel Size (Parameters)=9.41B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=112, Speed (words per second)=144k2022.039.84
Switch Transformer - EnlargeModel Size (Parameters)=10.31B, Training Steps=100k, FLOPs=2.3T, Time to Quality (hours)=140, Speed (words per second)=126k2022.0310.09
Switch TransformerModel Size (Parameters)=9.03B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=120, Speed (words per second)=139k2022.0310.45
sMLP - deterministicModel Size (Parameters)=3.50B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=20.41, Speed (words per second)=192k2022.0313.46
HASH LayersModel Size (Parameters)=3.51B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=21.08, Speed (words per second)=176k2022.0313.57
GshardModel Size (Parameters)=9.03B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=198, Speed (words per second)=115k2022.0314.5
Switch TransformerModel Size (Parameters)=3.48B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=20.67, Speed (words per second)=185k2022.0315.31
STABLEMOESize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.0416.22
BASE LayerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.0416.36
Hash LayerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.0416.37
Switch TransformerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.0416.62
HASH LayersModel Size (Parameters)=10.29B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=100, Speed (words per second)=141k2022.0316.69
Larger Transformer (deeper)Size=Base, # Shared Params=578M, # Expert Params=N/A, FLOPs=610B2022.0417.93
Base LayersModel Size (Parameters)=3.60B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=21.13, Speed (words per second)=178k2022.0318.16
Larger Transformer (wider)Size=Base, # Shared Params=578M, # Expert Params=N/A, FLOPs=610B2022.0418.31
Standard TransformerSize=Large, # Shared Params=355M, # Expert Params=N/A, FLOPs=414B2022.0418.86
GshardModel Size (Parameters)=3.48B, Training Steps=25k, FLOPs=0.8T, Time to Quality (hours)=27.54, Speed (words per second)=136k2022.0319.26
STABLEMOESize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.0419.28
Hash LayerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.0419.63
Switch TransformerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.0419.79
BASE LayerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.0420.04
Base LayersModel Size (Parameters)=10.31B, Training Steps=100k, FLOPs=2.0T, Time to Quality (hours)=115.13, Speed (words per second)=135k, note=divergence during training2022.0320.36
Standard TransformerSize=Base, # Shared Params=124M, # Expert Params=N/A, FLOPs=146B2022.0423.02
Hash LayerConfiguration=layers=11, modules=1x64, Params=794M2021.0626.99
Switch TransformerConfiguration=layers=11, modules=1x64, load_bal=0.1, Params=795M2021.0627.41
Baseline TransformerConfiguration=layers=11, d=1024, D=4096, Params=266M2021.0628.85