Language Modeling on LLM (val)
1.3364LossdynActGLU(Swish)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| dynActGLU(Swish)Activation=dynActGLU(Swish), Training steps=343002026.03 | 1.3364 | 3.8056 | |
| Baseline SwiGLUActivation=SwiGLU, Training steps=343002026.03 | 1.3374 | 3.8091 | |
| dynActGLU(Swish)Activation=dynActGLU(Swish), Training steps=56202026.03 | 1.3979 | 4.0467 | |
| Baseline SwiGLUActivation=Baseline SwiGLU, Training steps=56202026.03 | 1.5072 | 4.5142 |