Language Modeling on OpenWebText2 (val)
17.12PerplexityDenseFormer
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DenseFormerDilation × DWA Period=1 × 1, Depth=72, Parameters # (M)=548.362024.02 | 17.12 | 2.93 | |
| DenseFormerDilation × DWA Period=4 × 1, Depth=72, Parameters # (M)=548.352024.02 | 17.17 | 3.6 | |
| DenseFormerDilation × DWA Period=4 × 5, Depth=72, Parameters # (M)=548.352024.02 | 17.21 | 3.9 | |
| TransformerDepth=90, Parameters # (M)=675.782024.02 | 17.44 | 3.32 | |
| TransformerDepth=84, Parameters # (M)=633.312024.02 | 17.48 | 3.54 | |
| TransformerDepth=72, Parameters # (M)=548.352024.02 | 17.82 | 4.08 | |
| DenseFormerDilation × DWA Period=1 × 1, Depth=48, Parameters # (M)=378.452024.02 | 17.84 | 4.65 | |
| DenseFormerDilation × DWA Period=4 × 1, Depth=48, Parameters # (M)=378.452024.02 | 17.86 | 5.31 | |
| DenseFormerDilation × DWA Period=4 × 5, Depth=48, Parameters # (M)=378.452024.02 | 17.87 | 5.72 | |
| TransformerDepth=64, Parameters # (M)=491.722024.02 | 17.94 | 4.57 | |
| Skips With GainsDepth=48, Parameters # (M)=378.452024.02 | 18.45 | 5.72 | |
| TransformerDepth=48, Parameters # (M)=378.452024.02 | 18.61 | 5.94 |