Long sequence classification on LRA (test)
87.09Average AccuracyFull model
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Full modelBackbone=S52024.11 | 87.09 | 61.48 | 88.88 | 91.2 | 87.3 | 95.15 | 98.41 | |
| LASTBackbone=S52024.11 | 86.53 | 61.48 | 88.52 | 90.42 | 86.34 | 94.45 | 97.95 | |
| S4 (updated)state size N=642021.10 | 86.09 | 59.6 | 86.82 | 90.9 | 88.65 | 94.2 | 96.35 | |
| Paramixer (CDIL)Sequence Length=40002022.04 | 83.32 | — | — | — | — | — | — | |
| Paramixer (CHORD)Sequence Length=40002022.04 | 83.12 | — | — | — | — | — | — | |
| Full modelBackbone=S4D2024.11 | 81.05 | 56.42 | 86.4 | 90.46 | 77.02 | 87.94 | 88.07 | |
| LASTBackbone=S4D2024.11 | 80.77 | 56.27 | 85.95 | 89.46 | 77.02 | 87.83 | 88.07 | |
| Uniform H∞Backbone=S4D2024.11 | 80.73 | 55.82 | 86.02 | 89.87 | 77.02 | 87.59 | 88.07 | |
| S4 (original)state size N=2562021.10 | 80.48 | 58.35 | 76.02 | 87.09 | 87.26 | 88.05 | 88.1 | |
| Global H∞Backbone=S4D2024.11 | 79.71 | 49.95 | 86.2 | 89.84 | 77.02 | 87.2 | 88.07 | |
| Global H∞Backbone=S52024.11 | 75.74 | 61.48 | 88.56 | 90.93 | 87.04 | 57.2 | 69.21 | |
| Uniform H∞Backbone=S52024.11 | 75.5 | 61.48 | 82.49 | 90.29 | 86.45 | 71.38 | 90.9 | |
| Transformer-LSReference Source=[25], Sequence Length=40002022.04 | 68.4 | — | — | — | — | — | — | |
| Fourier Transformercompression_budget=80% reduction at first layer2023.05 | 67.54 | 40.73 | 75.02 | 85.35 | 53.17 | 83.43 | — | |
| RFA-GaussianReference Source=[18], Sequence Length=40002022.04 | 66 | — | — | — | — | — | — | |
| Linear TransformerReference Source=[24], Sequence Length=40002022.04 | 65.9 | — | — | — | — | — | — | |
| NyströmformerReference Source=[25], Sequence Length=40002022.04 | 65.75 | — | — | — | — | — | — | |
| NyströmformerReference Source=[15], Sequence Length=40002022.04 | 65.52 | — | — | — | — | — | — | |
| PerformerReference Source=[24], Sequence Length=40002022.04 | 65.4 | — | — | — | — | — | — | |
| TransformerReference Source=[25], Sequence Length=40002022.04 | 65.35 | — | — | — | — | — | — | |
| PerformerReference Source=[25], Sequence Length=40002022.04 | 65.21 | — | — | — | — | — | — | |
| TransformerReference Source=[15], Sequence Length=40002022.04 | 65.02 | — | — | — | — | — | — | |
| ReformerReference Source=[25], Sequence Length=40002022.04 | 64.88 | — | — | — | — | — | — | |
| ReformerReference Source=[15], Sequence Length=40002022.04 | 64.88 | — | — | — | — | — | — | |
| TransformerReference Source=[24], Sequence Length=40002022.04 | 64.27 | — | — | — | — | — | — | |
| FSATconvolutions_on_text=none2023.05 | 64.24 | 46.85 | 65.95 | 81.11 | 49.97 | 77.32 | — | |
| BigBirdReference Source=[24], Sequence Length=40002022.04 | 64.02 | — | — | — | — | — | — | |
| PerformerReference Source=[15], Sequence Length=40002022.04 | 63.81 | — | — | — | — | — | — | |
| Sparse TransformerReference Source=[24], Sequence Length=40002022.04 | 63.58 | — | — | — | — | — | — | |
| LongformerReference Source=[24], Sequence Length=40002022.04 | 62.58 | — | — | — | — | — | — | |
| Luna-256Sequence Length=2562023.05 | 61.24 | 37.25 | 64.57 | 79.29 | 47.38 | 77.32 | — | |
| FMMformerKernels=2, Bandwidth=52021.08 | 60.74 | 36.74 | 67.84 | 81.88 | 45.1 | 72.12 | — | |
| Flexformer_n2026.06 | 59.99 | 37.18 | 65.72 | 73.94 | 45.42 | 77.66 | — | |
| FMMformerKernels=1, Bandwidth=52021.08 | 59.51 | 33.22 | 66.52 | 81.5 | 45.01 | 71.29 | — | |
| Skyformer2021.10 | 59.39 | 38.69 | 64.7 | 82.06 | 40.77 | 70.73 | — | |
| Luna-2562021.10 | 59.37 | 37.25 | 64.57 | 79.29 | 47.38 | 77.72 | — | |
| Flexformer_stype=stationary variant2026.06 | 59.11 | 37.09 | 67.38 | 71.55 | 43.39 | 76.15 | — | |
| BigBird2021.10 | 59.05 | 39.25 | 63.86 | 80.28 | 43.16 | 68.72 | — | |
| Reformer2021.10 | 58.99 | 37.68 | 62.93 | 78.99 | 48.87 | 66.49 | — | |
| Nyströmformerimplementation=PyTorch, layers=2, embedding dimension=64, hidden dimension=128, attention heads=2, pooling=mean2021.02 | 58.95 | 37.15 | 65.52 | 79.56 | 41.58 | 70.94 | — | |
| Nyström2023.05 | 58.95 | 37.15 | 65.52 | 79.56 | 41.58 | 70.94 | — | |
| Nystromformer2021.10 | 58.93 | 38.51 | 64.83 | 80.52 | 41.3 | 69.48 | — | |
| Standardimplementation=PyTorch, layers=2, embedding dimension=64, hidden dimension=128, attention heads=2, pooling=mean2021.02 | 58.77 | 37.1 | 65.02 | 79.35 | 38.2 | 74.16 | — | |
| SoftmaxAttention=Standard Softmax2021.08 | 58.7 | 37.1 | 64.17 | 80.71 | 39.06 | 72.48 | — | |
| Kernelized Attention2021.10 | 58.56 | 38.78 | 60.22 | 81.77 | 41.29 | 70.73 | — | |
| Combiner-Fixed2021.07 | 58.34 | 36.65 | 64.99 | 59.81 | 41.67 | 88.59 | — | |
| Performer2021.10 | 58 | 38.02 | 64.19 | 80.04 | 41.43 | 66.3 | — | |
| Transformer2021.07 | 57.87 | 36.38 | 64.27 | 57.46 | 42.44 | 88.81 | — | |
| Band5Attention=Softmax with banded attention, Bandwidth=52021.08 | 57.73 | 32.16 | 66.31 | 79.41 | 43.33 | 67.44 | — | |
| BigBird2021.07 | 57.59 | 37.08 | 64.02 | 59.29 | 40.83 | 86.75 | — | |
| Hedgehog2026.06 | 57.47 | 37.02 | 64.1 | 68.67 | 41.64 | 75.89 | — | |
| Nyströmformer2021.10 | 57.46 | 37.15 | 65.52 | 79.56 | 41.58 | 70.94 | — | |
| Sparse Trans.2021.07 | 57.42 | 35.78 | 63.58 | 59.59 | 44.24 | 83.9 | — | |
| Self-Attention2021.10 | 57.37 | 38.37 | 61.95 | 80.69 | 40.57 | 65.26 | — | |
| Performer2021.07 | 57.35 | 36 | 65.4 | 53.82 | 42.77 | 88.76 | — | |
| Combiner-Axial2021.07 | 57.27 | 36.15 | 64.36 | 56.1 | 41.33 | 88.43 | — | |
| Polaformer2026.06 | 57.23 | 36.93 | 69.65 | 65.39 | 40.95 | 73.26 | — | |
| Skyformer2026.06 | 57.16 | 36.71 | 65.51 | 68.05 | 41.36 | 74.18 | — | |
| Longformer2021.07 | 56.93 | 36.03 | 62.85 | 56.89 | 42.22 | 86.68 | — | |
| Transformer2026.06 | 56.55 | 37.23 | 65.66 | 64.49 | 41.89 | 73.46 | — | |
| Cosformer2026.06 | 56.5 | 37.44 | 65.16 | 64.04 | 43.56 | 72.32 | — | |
| BigBird2026.06 | 56.39 | 36.88 | 65.34 | 64.23 | 41.93 | 73.58 | — | |
| LinformerReference Source=[25], Sequence Length=40002022.04 | 56.12 | — | — | — | — | — | — | |
| ReformerReference Source=[24], Sequence Length=40002022.04 | 56.1 | — | — | — | — | — | — | |
| LinformerReference Source=[15], Sequence Length=40002022.04 | 55.91 | — | — | — | — | — | — | |
| Linformerimplementation=PyTorch, layers=2, embedding dimension=64, hidden dimension=128, attention heads=2, pooling=mean, projection dimension=2562021.02 | 55.59 | 37.25 | 55.91 | 79.37 | 37.84 | 67.6 | — | |
| FNet2023.05 | 55.3 | 35.55 | 65.11 | 59.61 | 38.67 | 77.8 | — | |
| Synthesizer2021.07 | 55.21 | 36.5 | 61.68 | 54.67 | 41.61 | 81.61 | — | |
| Reformerimplementation=PyTorch, layers=2, embedding dimension=64, hidden dimension=128, attention heads=2, pooling=mean, number of hashes=22021.02 | 55.04 | 19.05 | 64.88 | 78.64 | 43.29 | 69.36 | — | |
| BigBird2023.05 | 55.01 | 36.05 | 64.02 | 59.29 | 40.83 | 74.87 | — | |
| Linformer2021.10 | 54.69 | 37.45 | 58.93 | 78.19 | 37.96 | 60.93 | — | |
| LinearAttention=Linear2021.08 | 54.67 | 18.3 | 64.22 | 81.37 | 38.29 | 71.17 | — | |
| Longformer2026.06 | 54.49 | 36.46 | 63.09 | 59.24 | 41.23 | 72.44 | — | |
| FNet2021.10 | 54.42 | 35.33 | 65.11 | 59.61 | 38.67 | 77.8 | — | |
| Transformer2023.05 | 54.39 | 36.37 | 64.27 | 57.46 | 42.44 | 71.4 | — | |
| Reformer2026.06 | 54.28 | 36.62 | 63.52 | 57.6 | 40.06 | 73.61 | — | |
| BigBird2021.10 | 54.17 | 36.05 | 64.02 | 59.29 | 40.83 | 74.87 | — | |
| LinformerReference Source=[24], Sequence Length=40002022.04 | 53.94 | — | — | — | — | — | — | |
| Informer2021.10 | 53.73 | 32.53 | 62.64 | 77.57 | 38.1 | 57.83 | — | |
| Transformerlayers=4, hidden dimension=256, heads=42021.10 | 53.66 | 36.37 | 64.27 | 57.46 | 42.44 | 71.4 | — | |
| Performerimplementation=PyTorch, layers=2, embedding dimension=64, hidden dimension=128, attention heads=2, pooling=mean, random feature dimension=2562021.02 | 53.63 | 18.8 | 63.81 | 78.62 | 37.07 | 69.87 | — | |
| Longformer2023.05 | 53.46 | 35.63 | 62.85 | 56.89 | 42.22 | 69.71 | — | |
| Linear Trans.2021.07 | 53.32 | 17.15 | 65.9 | 53.09 | 42.34 | 88.13 | — | |
| Linformer2021.07 | 53.28 | 35.49 | 53.94 | 52.27 | 38.56 | 86.17 | — | |
| Longformer2021.10 | 52.88 | 35.63 | 62.85 | 56.89 | 42.22 | 69.71 | — | |
| Synthesizer2023.05 | 52.88 | 36.99 | 61.68 | 54.67 | 41.61 | 69.45 | — | |
| Sinkhorn Trans.2021.07 | 52.76 | 34.2 | 61.2 | 53.83 | 41.23 | 73.36 | — | |
| Reformer2021.07 | 52.61 | 36.3 | 56.1 | 53.4 | 38.07 | 79.18 | — | |
| Synthesizer2021.10 | 52.4 | 36.99 | 61.68 | 54.67 | 41.61 | 69.45 | — | |
| Linear Trans.2026.06 | 51.49 | 17.42 | 62.64 | 58.92 | 43.4 | 75.11 | — | |
| Performer2023.05 | 51.41 | 18.01 | 65.4 | 53.82 | 42.77 | 77.5 | — | |
| Linformer2023.05 | 51.36 | 35.7 | 53.94 | 52.27 | 38.56 | 76.34 | — | |
| RFA2026.06 | 51.27 | 17.2 | 64.89 | 58.76 | 41 | 74.51 | — | |
| Sinkhorn Trans.2021.10 | 51.23 | 33.67 | 61.2 | 53.83 | 41.23 | 67.45 | — | |
| Performer2021.10 | 51.18 | 18.01 | 65.4 | 53.82 | 42.77 | 77.05 | — | |
| Linformer2021.10 | 51.14 | 35.7 | 53.94 | 52.27 | 38.56 | 76.34 | — | |
| Sparse Trans.2021.10 | 51.03 | 17.07 | 63.58 | 59.59 | 44.24 | 71.71 | — | |
| Reformer2023.05 | 50.67 | 37.27 | 56.1 | 53.4 | 38.07 | 68.5 | — | |
| Reformer2021.10 | 50.56 | 37.27 | 56.1 | 53.4 | 38.07 | 68.5 | — | |
| Performer2026.06 | 50.48 | 16.97 | 63.67 | 55.17 | 41.23 | 75.38 | — |