Language Modeling on LM1B ctx len. 128 (val)
25.72PPL (Val)MDLM† (Compl. masking)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MDLM† (Compl. masking)#Params=170M, Batch size=32, Context length=128, Training batch size=2x larger2025.05 | 25.72 | 3.78 | 1,081.57 | |
| PGM 6 / 6#Params=171M, Batch size=32, Context length=128, Encoder layers=6, Decoder layers=62025.05 | 26.8 | 2.12 | 1,930.93 | |
| MDLM#Params=170M, Batch size=32, Context length=1282025.05 | 27.67 | 3.78 | 1,081.57 |