Language Modeling on Text8 (val)
0.9305Validation LossWAV v1
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| WAV v1Depth=482026.06 | 0.9305 | -0.0057 | 0.57 | |
| Block AttnResDepth=482026.06 | 0.9363 | — | — | |
| StandardDepth=482026.06 | 0.9543 | — | — | |
| Block AttnResDepth=242026.06 | 0.9592 | — | — | |
| WAV v1Depth=242026.06 | 0.961 | 0.0018 | -0.18 | |
| StandardDepth=242026.06 | 0.9736 | — | — | |
| Block AttnResDepth=122026.06 | 0.9801 | — | — | |
| LayerScaleDepth=482026.06 | 0.9919 | — | — | |
| StandardDepth=122026.06 | 0.9955 | — | — | |
| ReZeroDepth=482026.06 | 1.0003 | — | — | |
| LayerScaleDepth=122026.06 | 1.0057 | — | — | |
| LayerScaleDepth=242026.06 | 1.0068 | — | — | |
| WAV v1Depth=122026.06 | 1.0085 | 0.0284 | -2.88 | |
| ReZeroDepth=242026.06 | 1.0183 | — | — | |
| ReZeroDepth=122026.06 | 1.0404 | — | — |