Language Modeling on RoBERTa+cc100en (test)
15.59PPLSTABLEMOE
Evaluation results
| Method | Links | |
|---|---|---|
| STABLEMOESize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 15.59 | |
| BASE LayerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 15.75 | |
| Hash LayerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 15.79 | |
| Switch TransformerSize=Large, # Shared Params=355M, # Expert Params=3.22B, FLOPs=465B2022.04 | 16.21 | |
| Larger Transformer (deeper)Size=Base, # Shared Params=578M, # Expert Params=N/A, FLOPs=610B2022.04 | 17.63 | |
| Larger Transformer (wider)Size=Base, # Shared Params=578M, # Expert Params=N/A, FLOPs=610B2022.04 | 18.01 | |
| Standard TransformerSize=Large, # Shared Params=355M, # Expert Params=N/A, FLOPs=414B2022.04 | 18.19 | |
| STABLEMOESize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 18.93 | |
| Switch TransformerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 19.2 | |
| Hash LayerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 19.25 | |
| BASE LayerSize=Base, # Shared Params=124M, # Expert Params=454M, FLOPs=160B2022.04 | 19.69 | |
| Standard TransformerSize=Base, # Shared Params=124M, # Expert Params=N/A, FLOPs=146B2022.04 | 22.58 |