Masked Language Modeling on WikiText-103 (val)
6.6764Validation LossGEM (N = 1)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GEM (N = 1)Architecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,427 tok/s, Latency=42.51 ms/step2026.04 | 6.6764 | 0.034 | 0.012 | |
| GELU (tanh)Architecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,998 tok/s, Latency=41.16 ms/step2026.04 | 6.6815 | 0.03 | 0.007 | |
| GEM (N = 2)Architecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=20,022 tok/s, Latency=41.16 ms/step2026.04 | 6.6819 | 0.036 | 0.006 | |
| GELUArchitecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,142 tok/s, Latency=43.03 ms/step2026.04 | 6.688 | 0.04 | — | |
| ReLUArchitecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=18,763 tok/s, Latency=43.71 ms/step2026.04 | 6.6899 | 0.045 | 0.002 | |
| SiLU/SwishArchitecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,359 tok/s, Latency=42.55 ms/step2026.04 | 6.6962 | 0.026 | 0.008 |