HMM Symbol Prediction on HMM Easy v1 (val)
2.7Excess over BayesTwo-layer causal transformer
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Two-layer causal transformerWidth=128, Attention heads=4, MLP width=256, Parameters=275k, Sequence length=64, Training sequences=60002026.05 | 2.7 | 0.772 | 41.4 | 0.265 | 0.056 | 0.016 |