Language Modeling on OpenWebText ctx len. 1024 (val)
21.43PerplexityPGM 6 / 6 (dim. 1024)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| PGM 6 / 6 (dim. 1024)#Params=268M, Batch size=32, Context length=1024, Encoder layers=6, Decoder layers=6, Model dimension=10242025.05 | 21.43 | 5.93 | 5,518.09 | |
| PGM 8 / 8#Params=203M, Batch size=32, Context length=1024, Encoder layers=8, Decoder layers=82025.05 | 22.61 | 5.86 | 5,585.57 | |
| MDLM† (Compl. masking)#Params=170M, Batch size=32, Context length=1024, Training batch size=2x larger2025.05 | 22.98 | 31.41 | 1,043.22 | |
| MDLM#Params=170M, Batch size=32, Context length=10242025.05 | 23.07 | 31.41 | 1,043.22 |