Algorithmic Reasoning Suite Unseen Length (test)
100Even Pairssin/cos (Randomized)
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sin/cos (Randomized)Positional Encoding=sin/cos, Randomization Strategy=Randomized2023.05 | 100 | 25.7 | 52.6 | 59 | 72.8 | 75.6 | 33.8 | 24.5 | 72.4 | 52.5 | 65.9 | 64.4 | 52.1 | 52.5 | 100 | |
| Relative (Randomized)Positional Encoding=Relative, Randomization Strategy=Randomized2023.05 | 100 | 28.1 | 52.2 | 58.8 | 77.9 | 95.1 | 34.9 | 28.1 | 75.1 | 100 | 69.3 | 64.5 | 50.1 | 53.3 | 100 | |
| RoPE (Randomized)Positional Encoding=RoPE, Randomization Strategy=Randomized2023.05 | 100 | 25.5 | 52.3 | 73.6 | 68.2 | 69.9 | 32.7 | 24.5 | 68.9 | 88.7 | 65.6 | 60.2 | 51.7 | 52.3 | 99.6 | |
| Learned* (Randomized)Positional Encoding=Learned*, Randomization Strategy=Randomized2023.05 | 97.5 | 21.1 | 52.6 | 49.7 | 69.1 | 52.9 | 31.9 | 22.1 | 53 | 52.7 | 52.7 | 61.7 | 51.9 | 52 | 99.5 | |
| RelativePositional Encoding=Relative, Randomization Strategy=Standard2023.05 | 96.4 | 21.8 | 51.8 | 23 | 53.6 | 58.3 | 30.3 | 23 | 51.7 | 54 | 52.7 | 54.3 | 52.2 | 52.4 | 91.9 | |
| ALiBi (Randomized)Positional Encoding=ALiBi, Randomization Strategy=Randomized2023.05 | 81.5 | 21.2 | 50.3 | 29.8 | 70.6 | 77.1 | 31.3 | 22 | 68.9 | 79.7 | 64.7 | 56.2 | 50.5 | 51.2 | 99.6 | |
| ALiBiPositional Encoding=ALiBi, Randomization Strategy=Standard2023.05 | 67.3 | 24.2 | 51.7 | 37.6 | 57.5 | 62.3 | 32.5 | 25.7 | 51.3 | 54.3 | 51.4 | 51.4 | 51 | 50.9 | 38.8 | |
| RoPEPositional Encoding=RoPE, Randomization Strategy=Standard2023.05 | 51 | 21.6 | 51.3 | 23.6 | 51.2 | 51.9 | 25.1 | 23.1 | 50.9 | 56.5 | 51.3 | 50.4 | 50.2 | 50.5 | 30.6 | |
| sin/cosPositional Encoding=sin/cos, Randomization Strategy=Standard2023.05 | 50.9 | 20.5 | 50.5 | 26.3 | 50.1 | 50.6 | 28.3 | 21 | 50.7 | 51.3 | 51.6 | 49.8 | 50.1 | 50.1 | 30.1 | |
| LearnedPositional Encoding=Learned, Randomization Strategy=Standard2023.05 | 50.7 | 20.2 | 50.3 | 24.2 | 49.2 | 50.7 | 25.1 | 20.4 | 50.8 | 51 | 50.6 | 49.8 | 49.6 | 50.2 | 25.9 | |
| NonePositional Encoding=None, Randomization Strategy=None2023.05 | 50.4 | 20.1 | 51.9 | 61.9 | 50.3 | 52.8 | 31 | 20.1 | 52.8 | 52.5 | 52.8 | 50.1 | 49.9 | 50.2 | 23.7 |