Causal Language Modeling on WikiText-103 (val)
72.57Validation PerplexityGEM (N = 2)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GEM (N = 2)Activation=GEM (N = 2), Throughput (tok/s)=8,603, Standard Deviation=0.162026.04 | 72.57 | 4.2847 | 1.19 | |
| GEM (N = 1)Activation=GEM (N = 1), Throughput (tok/s)=8,604, Standard Deviation=0.132026.04 | 73.32 | 4.2948 | 0.44 | |
| GELUActivation=GELU, Throughput (tok/s)=8,601, Standard Deviation=0.302026.04 | 73.76 | 4.3008 | — | |
| GELU (tanh)Activation=GELU (tanh), Throughput (tok/s)=8,478, Standard Deviation=0.302026.04 | 73.76 | 4.3008 | 0 | |
| ReLUActivation=ReLU, Throughput (tok/s)=8,634, Standard Deviation=0.192026.04 | 77.64 | 4.3521 | 3.88 | |
| SiLU/SwishActivation=SiLU/Swish, Throughput (tok/s)=8,594, Standard Deviation=0.782026.04 | 82.53 | 4.4131 | 8.77 |