Masked Reconstruction on WikiText-103
4.94PPLWPT
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| WPTTotal Parameters (M)=31.8, No-Embedding Parameters (M)=15.0, Core Parameters (M)=12.92025.12 | 4.94 | 67.2 | 82.8 | |
| TRANSF.Total Parameters (M)=32.5, No-Embedding Parameters (M)=15.8, Core Parameters (M)=15.82025.12 | 5.28 | 66.1 | 81.9 | |
| PRISM (H)Total Parameters (M)=32.9, No-Embedding Parameters (M)=16.2, Core Parameters (M)=14.02025.12 | 6.06 | 64.3 | 80.3 | |
| HSSMTotal Parameters (M)=33.0, No-Embedding Parameters (M)=15.1, Core Parameters (M)=13.02025.12 | 6.47 | 63.2 | 79.5 | |
| FNET (H)Total Parameters (M)=34.7, No-Embedding Parameters (M)=15.8, Core Parameters (M)=15.82025.12 | 9.87 | 56.1 | 74.2 |