Language Modeling on peS2o
9.78PPLTransformer
Evaluation results
| Method | Links | |
|---|---|---|
| Transformertotal_params=262M, n_heads=16, MACs=5.4G, Mem (floats)=21M2023.12 | 9.78 | |
| SwitchAlltotal_params=262M, n_heads=4, MACs=2.4G, Mem (floats)=5.6M2023.12 | 9.86 | |
| SwitchAlltotal_params=47M, n_heads=2, MACs=202M, Mem (floats)=0.8M2023.12 | 12.56 | |
| Transformertotal_params=47M, n_heads=10, MACs=453M, Mem (floats)=3.5M2023.12 | 12.83 |