List your company
ResearchFormerly WizWand

Masked Language Modeling on WikiText-103 (train)

6.6188Training LossGEM (N = 1)
6.6178086.6245046.63126.637896Apr 23, 2026
Updated 6mo ago

Evaluation results

MethodLinks
GEM (N = 1)Architecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,427 tok/s, Latency=42.51 ms/step2026.046.6188
GEM (N = 2)Architecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=20,022 tok/s, Latency=41.16 ms/step2026.046.6267
GELU (tanh)Architecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,998 tok/s, Latency=41.16 ms/step2026.046.628
GELUArchitecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,142 tok/s, Latency=43.03 ms/step2026.046.6341
ReLUArchitecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=18,763 tok/s, Latency=43.71 ms/step2026.046.6413
SiLU/SwishArchitecture=BERT-small, Number of parameters=28.6M, Training steps=3,000, Seeds=3, Throughput=19,359 tok/s, Latency=42.55 ms/step2026.046.6436