Long-range Sequence Modeling on Long Range Arena (LRA)
90.53Text AccuracyMEGALODON
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| MEGALODONMode=chunk2024.04 | 90.53 | 87.11 | 96.89 | 97.21 | 62.23 | 91.74 | 87.62 | — | |
| MEGALODONMode=full2024.04 | 90.48 | 89.42 | 98.13 | 98.17 | 63.79 | 91.76 | 88.63 | — | |
| Mega# parameters=~600k2023.06 | 90.43 | 90.44 | 96.01 | 97.98 | 63.14 | 91.25 | — | — | |
| MEGAMode=full2024.04 | 90.43 | 90.44 | 96.01 | 97.98 | 63.14 | 91.25 | 88.21 | — | |
| Mega2025.12 | 90.43 | 90.44 | 96.01 | 97.98 | — | 91.25 | 93.22 | — | |
| Mega2023.11 | 90.43 | 90.44 | 96.01 | 97.98 | 63.14 | 91.25 | 88.21 | — | |
| Mega + SMRMode=Convolution, SMR=true2024.05 | 90.36 | 88.21 | 93.78 | — | 59.45 | 90.64 | 84.49 | — | |
| MEGAMode=chunk2024.04 | 90.19 | 85.8 | 94.41 | 93.81 | 58.76 | 90.97 | 85.66 | — | |
| MegaMode=Convolution, SMR=false2024.05 | 89.97 | 86.82 | 93.4 | — | 57.67 | 90.17 | 83.61 | — | |
| LRU2025.12 | 89.4 | 89 | 95.1 | 94.2 | — | 89.9 | 91.52 | — | |
| LRU2023.11 | 89.4 | 89 | 95.1 | 94.2 | 60.2 | 89.9 | 86.3 | — | |
| S52025.12 | 89.31 | 88 | 95.33 | 98.56 | — | 91.4 | 92.52 | — | |
| S52023.11 | 89.31 | 88 | 95.33 | 98.56 | 62.15 | 91.4 | 87.46 | — | |
| SGConv2025.12 | 89.2 | 87.97 | 95.46 | 97.83 | — | 91.11 | 92.31 | — | |
| SGConv2023.11 | 89.2 | 87.97 | 95.46 | 97.83 | 61.45 | 91.11 | 87.17 | — | |
| S4 + SMRMode=Convolution, SMR=true2024.05 | 89.09 | 88.97 | 89.01 | — | 59.01 | 89.28 | 83.07 | — | |
| HGRN2023.11 | 88.14 | 88.69 | 92.92 | 97.5 | 59.95 | 94.23 | 86.91 | — | |
| TNN2025.12 | 87.9 | 88.24 | 93 | 96.1 | — | 90.97 | 91.24 | — | |
| TNN2023.11 | 87.9 | 88.24 | 93 | 96.1 | 61.04 | 90.97 | 86.21 | — | |
| CDIL-CNNChannels=642022.01 | 87.61 | 64.49 | 91 | — | — | 84.27 | — | — | |
| CNNChannels=642022.01 | 87.29 | 35.85 | 55.95 | — | — | 83.33 | — | — | |
| MS-SSMBackbone=S42025.12 | 87.22 | 89.15 | 94.9 | 97.12 | — | 91.06 | 91.89 | — | |
| S5 + SMRMode=Recurrence, SMR=true2024.05 | 87.08 | 89.31 | 88.05 | — | 59.08 | 89.37 | 82.58 | — | |
| SPADE + SMRMode=Convolution, SMR=true2024.05 | 87.06 | 89.29 | 93.34 | — | 59.52 | 89.01 | 83.64 | — | |
| Deformable2022.01 | 86.91 | 36.57 | 56.14 | — | — | 83.69 | — | — | |
| S42025.12 | 86.82 | 88.65 | 94.2 | 96.35 | — | 90.9 | 91.38 | — | |
| S42023.11 | 86.82 | 88.65 | 94.2 | 96.35 | 59.6 | 90.9 | 86.09 | — | |
| S4Streaming?=No2026.02 | 86.82 | 88.65 | 94.2 | — | 59.6 | 90.9 | 86.09 | — | |
| GRUHidden size=128, Layers=12022.01 | 86.7 | 44.4 | 85.45 | — | — | 77.08 | — | — | |
| S42024.04 | 86.53 | 88.48 | 94.01 | 96.07 | 59.1 | 90.94 | 85.86 | — | |
| S4Mode=Convolution, SMR=false2024.05 | 86.47 | 87.2 | 85.99 | — | 57.06 | 86.74 | 80.69 | — | |
| SPADEMode=Convolution, SMR=false2024.05 | 86.29 | 88.05 | 92.77 | — | 58.75 | 88.62 | 82.9 | — | |
| LSTMHidden size=128, Layers=12022.01 | 85.8 | 32.99 | 61.26 | — | — | 77.18 | — | — | |
| MS-SSMBackbone=S62025.12 | 85.7 | 89.83 | 87.24 | 87.7 | — | 83.21 | 86.73 | — | |
| DSS softmax2025.12 | 84.8 | 85.7 | 84.6 | 87.8 | — | 87.8 | 86.13 | — | |
| DSS_softmax2023.11 | 84.8 | 85.7 | 84.6 | 87.8 | 60.6 | 87.8 | 81.88 | — | |
| DSSEXP2025.12 | 84.6 | 84.9 | 84.7 | 85.6 | — | 87.6 | 85.47 | — | |
| DSSEXP2023.11 | 84.6 | 84.9 | 84.7 | 85.6 | 59.7 | 87.6 | 81.18 | — | |
| S5Mode=Recurrence, SMR=false2024.05 | 84.2 | 87.51 | 87.42 | — | 58.25 | 87.99 | 81.07 | — | |
| S6 + SMRMode=Recurrence, SMR=true2024.05 | 84.14 | 83.23 | 85.8 | — | 56.15 | 83.66 | 78.6 | — | |
| S6Mode=Recurrence, SMR=false2024.05 | 83.52 | 82.96 | 85.54 | — | 55.62 | 83.28 | 78.18 | — | |
| Mamba2025.12 | 82.98 | 69.82 | 69.26 | 67.32 | — | 72.14 | 72.3 | — | |
| DSSEXPScaling=No2025.12 | 82.4 | 81.2 | 81.3 | — | — | 86 | 66.46 | — | |
| DSSEXP-NO-SCALE2023.11 | 82.4 | 81.2 | 81.3 | — | 59.3 | 86 | 65.03 | — | |
| ELM Neuron# parameters=~100k-200k2023.06 | 80.3 | 49.62 | 71.15 | 77.29 | 46.77 | 84.93 | — | — | |
| H-Transformer-1D2021.07 | 78.69 | 46.05 | 68.78 | — | 49.53 | 63.99 | 61.41 | — | |
| S4# parameters=~600k2023.06 | 76.02 | 87.26 | 86.05 | 88.1 | 58.35 | 87.09 | — | — | |
| Chr.-LSTM# parameters=~100k-200k2023.06 | 75.4 | 46.09 | 70.79 | — | 44.55 | 82.87 | — | — | |
| LUNA2025.12 | 73.41 | 64.32 | 69.52 | — | 38.94 | 81.02 | 65.44 | — | |
| PolaFormeralpha=32025.12 | 73.06 | 42.15 | 70.53 | — | 37.35 | 80.5 | 60.72 | — | |
| Griffin2025.12 | 71.75 | 61.15 | 73.38 | 69.53 | — | 66.58 | 68.47 | — | |
| LOTFormer2025.12 | 71.1 | 54.1 | 69.9 | — | 38.5 | 80.9 | 62.9 | — | |
| Transformer-LSReference=[54]2022.01 | 68.4 | — | — | — | — | 81.95 | — | — | |
| cosFormer2025.12 | 67.7 | 51.23 | 71.96 | — | — | 83.15 | 68.51 | — | |
| cosFormer2023.11 | 67.7 | 51.23 | 71.96 | — | 36.5 | 83.15 | 51.76 | — | |
| Transformer-LSMode=None, SMR=false2024.05 | 66.62 | 69.98 | 47.6 | — | 40.3 | 81.68 | 61.24 | — | |
| RFA-GaussianReference=[25]2022.01 | 66 | — | — | — | — | 56.1 | — | — | |
| Linear Trans.2021.07 | 65.9 | 42.34 | 75.3 | — | 16.13 | 53.09 | 50.55 | — | |
| Linear Trans.2022.02 | 65.9 | 42.34 | 75.3 | — | 16.13 | 53.09 | 50.55 | — | |
| Linear TransformerReference=[53]2022.01 | 65.9 | 42.34 | 75.3 | — | — | 53.09 | — | — | |
| Linear Trans.2022.02 | 65.9 | 42.34 | 75.3 | — | 16.13 | 53.09 | 50.55 | — | |
| LinearTrans.2025.12 | 65.9 | 42.34 | 75.3 | — | 16.13 | 53.09 | 50.55 | — | |
| Linear Trans.Streaming?=No2026.02 | 65.9 | 42.34 | 75.3 | — | 16.13 | 53.09 | 50.46 | — | |
| Luna-2562024.04 | 65.78 | 47.86 | 78.55 | — | 37.98 | 79.56 | 61.95 | — | |
| NyströmformerReference=[54]2022.01 | 65.75 | — | — | — | — | 81.29 | — | — | |
| OmniNetAttention Kernel=Performer, Configuration Group=LRA Baselines2021.03 | 65.6 | — | — | — | 18.2 | 60.9 | 48.2 | — | |
| NyströmformerReference=[55]2022.01 | 65.52 | 41.58 | 70.94 | — | — | 79.56 | — | — | |
| NyströmformerStreaming?=No2026.02 | 65.52 | 41.58 | 70.94 | — | 37.15 | 79.56 | 57.46 | — | |
| Performer2021.03 | 65.4 | — | — | — | 18 | 53.8 | 45.7 | — | |
| Performer2021.07 | 65.4 | 42.77 | 77.05 | — | 18.01 | 53.82 | 51.41 | — | |
| Performer2022.02 | 65.4 | 42.77 | 77.05 | — | 18.01 | 53.82 | 51.41 | — | |
| PerformerReference=[53]2022.01 | 65.4 | 42.77 | 77.05 | — | — | 53.82 | — | — | |
| Performer2022.02 | 65.4 | 42.77 | 77.05 | — | 18.01 | 53.82 | 51.41 | — | |
| Performer2025.12 | 65.4 | 42.77 | 77.05 | — | 18.01 | 53.82 | 51.41 | — | |
| PerformerStreaming?=No2026.02 | 65.4 | 42.77 | 77.05 | — | 18.01 | 53.82 | 51.18 | — | |
| TransformerReference=[54]2022.01 | 65.35 | — | — | — | — | 82.3 | — | — | |
| PerformerReference=[54]2022.01 | 65.21 | — | — | — | — | 81.7 | — | — | |
| Transformer2024.04 | 65.21 | 42.94 | 71.83 | — | 37.11 | 79.14 | 59.24 | — | |
| FNetStreaming?=No2026.02 | 65.11 | 38.67 | 77.8 | — | 35.33 | 59.61 | 54.42 | — | |
| OmniNetAttention Kernel=Performer, Configuration Group=Transformer Comparison2021.03 | 65.1 | — | — | — | 37.2 | 58.8 | 53.7 | — | |
| TransformerReference=[55]2022.01 | 65.02 | 38.2 | 74.16 | — | — | 79.35 | — | — | |
| ReformerReference=[55]2022.01 | 64.88 | 43.29 | 69.36 | — | — | 78.64 | — | — | |
| ReformerReference=[54]2022.01 | 64.88 | — | — | — | — | 78.64 | — | — | |
| Nyströmformer (Spec. cond.)Architecture=Spectral conditioned attention2026.03 | 64.8 | 40.2 | 73.7 | — | 37.8 | 80.6 | — | — | |
| Skyformer2025.12 | 64.7 | 40.77 | 70.73 | — | 38.69 | 82.06 | 59.39 | — | |
| Hedgehog2025.12 | 64.6 | 40.15 | 74.16 | — | 37.15 | 82.24 | 59.66 | — | |
| Luna-256Streaming?=No2026.02 | 64.57 | 47.38 | 77.72 | — | 37.25 | 79.29 | 59.37 | — | |
| ScatterbrainNumber of parameters=1/8 of full2021.10 | 64.55 | 43.65 | 69.91 | — | 38.6 | 80.22 | 59.38 | — | |
| ScatterbrainParameter Budget=1/8 of full2021.10 | 64.55 | 43.65 | 69.91 | — | 38.6 | 80.22 | 59.38 | — | |
| Orthoformer-64GFLOPS=0.24, Memory (MB)=3442021.06 | 64.42 | 33.26 | 66.41 | — | 33.87 | 78.36 | 55.26 | — | |
| suGRU(binary new)Streaming?=Yes2026.02 | 64.42 | 80.32 | 84.92 | — | 44.93 | 77.54 | 70.43 | — | |
| Flowformer2022.02 | 64.29 | 43.2 | 73.95 | — | 38.7 | 62.24 | 56.48 | — | |
| Transformer# parameters=~600k2023.06 | 64.27 | 42.44 | 71.4 | — | 36.37 | 57.46 | — | — | |
| Transformer2021.07 | 64.27 | 42.44 | 71.4 | — | 36.37 | 57.46 | 54.39 | — | |
| Transformer2022.02 | 64.27 | 42.44 | 71.4 | — | 36.37 | 57.46 | 54.39 | — | |
| TransformerReference=[53]2022.01 | 64.27 | 42.44 | 71.4 | — | — | 57.46 | — | — | |
| Transformer2022.02 | 64.27 | 42.44 | 71.4 | — | 36.37 | 57.46 | 54.39 | — | |
| TransformerStreaming?=No2026.02 | 64.27 | 42.44 | 71.4 | — | 36.37 | 57.46 | 53.66 | — | |
| Nyströmformer-128GFLOPS=0.62, Memory (MB)=7452021.06 | 64.17 | 36.16 | 52.32 | — | 36.9 | 78.67 | 53.64 | — | |
| suGRU(sigmoid)Streaming?=Yes2026.02 | 64.17 | 56.92 | 73.45 | — | 39.45 | 63.23 | 59.44 | — |