Causal Language Modeling on WikiText-103 GPT-2 (124M) (train)
4.4614Train LossGEM (N = 2)
Evaluation results
| Method | Links | |
|---|---|---|
| GEM (N = 2)Activation=GEM (N = 2), Throughput (tok/s)=8,6032026.04 | 4.4614 | |
| GEM (N = 1)Activation=GEM (N = 1), Throughput (tok/s)=8,6042026.04 | 4.4727 | |
| GELUActivation=GELU, Throughput (tok/s)=8,6012026.04 | 4.4833 | |
| GELU (tanh)Activation=GELU (tanh), Throughput (tok/s)=8,4782026.04 | 4.4833 | |
| ReLUActivation=ReLU, Throughput (tok/s)=8,6342026.04 | 4.5341 | |
| SiLU/SwishActivation=SiLU/Swish, Throughput (tok/s)=8,5942026.04 | 4.5967 |