HMM symbol prediction on HMM 6-state family v1 (val)
0.017Excess over BayesTwo-layer causal transformer
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Two-layer causal transformerWidth=128, Attention heads=4, MLP width=256, Parameters=275k, Sequence length=64, Training sequences=60002026.05 | 0.017 | 0.498 | 27.5 | 0.673 | 0.019 | 0.006 |