Causal Prediction on WikiText-2 (val)
5.4856Min Validation Loss[1,0,0,0]
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| [1,0,0,0]H×L=4×6, T=512, d=2562025.12 | 5.4856 | 0.562 | |
| [1,0,0,0]H×L=16×6, T=512, d=2562025.12 | 5.4918 | 0.108 | |
| QKV111H×L=16×6, T=512, d=2562025.12 | 5.4978 | 0 | |
| [1,1,1,1]H×L=4×6, T=512, d=2562025.12 | 5.5088 | 0.142 | |
| QKV111H×L=4×6, T=512, d=2562025.12 | 5.5167 | 0 | |
| [1,1,1,1]H×L=16×6, T=512, d=2562025.12 | 5.5179 | -0.364 | |
| [0,0,0,0]H×L=4×6, T=512, d=2562025.12 | 5.5524 | -0.648 | |
| [1,0,0,0]H×L=1×1, T=512, d=2562025.12 | 5.625 | 0.023 | |
| [1,1,1,1]H×L=1×1, T=512, d=2562025.12 | 5.6262 | 0.002 | |
| QKV111H×L=1×1, T=512, d=2562025.12 | 5.6263 | 0 | |
| [0,0,0,0]H×L=1×1, T=512, d=2562025.12 | 5.6264 | -0.001 |