Long-range Sequence Modeling on Long Range Arena (LRA) (test)
90.69Accuracy (Text)SPADE-Chunk
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SPADE-ChunkModel Type=Linear-Time Transformers2024.08 | 90.69 | — | — | — | 87.4 | — | — | — | — | — | — | — | — | 88.22 | 96.23 | 60.5 | 91.17 | — | — | 97.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPADE-ChunkCategory=Linear-Time Transformers2024.08 | 90.69 | — | — | — | 87.4 | — | — | — | — | — | — | — | — | 88.22 | 96.23 | 60.5 | 91.17 | — | — | 97.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MEGASpeed=2.9x, Mem.=0.31x2022.09 | 90.43 | — | — | — | 88.21 | — | — | — | — | — | — | — | — | 90.44 | 96.01 | 63.14 | 91.25 | 97.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MEGAInference Complexity=Quadratic, Hybrid=true2023.06 | 90.43 | — | — | — | 88.21 | — | — | — | — | — | — | — | — | 90.44 | 96.01 | 63.14 | 91.25 | — | — | 97.98 | 2.9 | 0.31 | — | — | — | — | — | — | — | — | — | — | |
| MEGASource=copied from [60]2023.11 | 90.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 63.14 | 91.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MEGAModel Type=Quadratic-Time Transformers2024.08 | 90.43 | — | — | — | 88.21 | — | — | — | — | — | — | — | — | 90.44 | 96.01 | 63.14 | 91.25 | — | — | 97.98 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MEGA-chunkchunk size=128 (Path-X: 4096), Speed=5.5x, Mem.=0.13x2022.09 | 90.19 | — | — | — | 85.66 | — | — | — | — | — | — | — | — | 85.8 | 94.41 | 58.76 | 90.97 | 93.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MEGA-chunkInference Complexity=Linear, Hybrid=true2023.06 | 90.19 | — | — | — | 85.66 | — | — | — | — | — | — | — | — | 85.8 | 94.41 | 58.76 | 90.97 | — | — | 93.81 | 7 | 0.09 | — | — | — | — | — | — | — | — | — | — | |
| MEGA-ChunkModel Type=Linear-Time Transformers2024.08 | 90.19 | — | — | — | 85.66 | — | — | — | — | — | — | — | — | 85.8 | 94.41 | 58.76 | 90.97 | — | — | 93.81 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MEGA-ChunkCategory=Linear-Time Transformers2024.08 | 90.19 | — | — | — | 85.66 | — | — | — | — | — | — | — | — | 85.8 | 94.41 | 58.76 | 90.97 | — | — | 93.81 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SeqBoat-fullInference Complexity=Sub-quadratic, Hybrid=true2023.06 | 89.6 | — | — | — | 87.33 | — | — | — | — | — | — | — | — | 89.96 | 95.87 | 61.65 | 91.67 | — | — | 95.28 | 6.2 | 0.07 | — | — | — | — | — | — | — | — | — | — | |
| SeqBoatInference Complexity=Linear, Hybrid=true2023.06 | 89.6 | — | — | — | 87.62 | — | — | — | — | — | — | — | — | 90.1 | 96.35 | 61.7 | 91.28 | — | — | 96.68 | 10.4 | 0.05 | — | — | — | — | — | — | — | — | — | — | |
| Long ConvModel Type=Convolutional Models2024.08 | 89.6 | — | — | — | 86.6 | — | — | — | — | — | — | — | — | 87 | 93.2 | 62.2 | 91.3 | — | — | 96 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Long ConvCategory=Convolutional Models2024.08 | 89.6 | — | — | — | 86.6 | — | — | — | — | — | — | — | — | 87 | 93.2 | 62.2 | 91.3 | — | — | 96 | — | — | — | — | — | — | — | — | — | — | — | — | |
| RotRNN2024.07 | 89.6 | — | — | — | 84.8 | — | — | — | — | — | — | — | — | 85.9 | 93 | 61.1 | 89.9 | — | — | 89.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-L, Fourier+SparseModel Type=Ablations2024.08 | 89.42 | — | — | — | 79.7 | — | — | — | — | — | — | — | — | 89.15 | 96.64 | 61.65 | 91.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-LModel Type=Ours2024.08 | 89.42 | — | — | — | 88.2 | — | — | — | — | — | — | — | — | 90.55 | 96.64 | 62.45 | 91.48 | — | — | 98.65 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-L, Fourier+SparseType=Ablation, Variant=Fourier+Sparse2024.08 | 89.42 | — | — | — | 79.7 | — | — | — | — | — | — | — | — | 89.15 | 96.64 | 61.65 | 91.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-LType=Ours2024.08 | 89.42 | — | — | — | 88.2 | — | — | — | — | — | — | — | — | 90.55 | 96.64 | 62.45 | 91.48 | — | — | 98.65 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRUImplementation Source=Original2023.03 | 89.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | 95.1 | 60.2 | 89.9 | — | 89 | 94.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-L, FourierModel Type=Ablations2024.08 | 89.4 | — | — | — | 87.84 | — | — | — | — | — | — | — | — | 89.3 | 95.75 | 62.45 | 91.48 | — | — | 98.65 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-L, FourierType=Ablation, Variant=Fourier2024.08 | 89.4 | — | — | — | 87.84 | — | — | — | — | — | — | — | — | 89.3 | 95.75 | 62.45 | 91.48 | — | — | 98.65 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRU2024.10 | 89.4 | — | — | — | — | — | — | — | — | — | — | — | — | 89 | 95.1 | 60.2 | 89.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRU2024.10 | 89.4 | — | — | — | — | — | — | — | — | — | — | — | — | 89 | 95.1 | 60.2 | 89.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRUsource=Axler, 20242024.07 | 89.4 | — | — | — | 86.3 | — | — | — | — | — | — | — | — | 89 | 95.1 | 60.2 | 89.9 | — | — | 94.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRUsource=Our Reprod.2024.07 | 89.4 | — | — | — | 84.1 | — | — | — | — | — | — | — | — | 85.2 | 90 | 57.9 | 89.4 | — | — | 92.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S5Inference Complexity=Linear, Hybrid=false2023.06 | 89.31 | — | — | — | 87.46 | — | — | — | — | — | — | — | — | 88 | 95.33 | 62.15 | 91.4 | — | — | 98.58 | 6.1 | 0.14 | — | — | — | — | — | — | — | — | — | — | |
| S5Model Type=State Space Models2024.08 | 89.31 | — | — | — | 87.46 | — | — | — | — | — | — | — | — | 88 | 95.33 | 62.15 | 91.4 | — | — | 98.58 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S5Category=State Space Models2024.08 | 89.31 | — | — | — | 87.46 | — | — | — | — | — | — | — | — | 88 | 95.33 | 62.15 | 91.4 | — | — | 98.58 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S52024.10 | 89.31 | — | — | — | — | — | — | — | — | — | — | — | — | 88 | 95.33 | 62.15 | 91.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S52024.10 | 89.31 | — | — | — | — | — | — | — | — | — | — | — | — | 88 | 95.33 | 62.15 | 91.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S5Implementation Source=Paper results2023.03 | 89.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | 95.3 | 62.2 | 91.4 | — | 90.1 | 98.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S52024.07 | 89.3 | — | — | — | 87.8 | — | — | — | — | — | — | — | — | 90.1 | 95.3 | 62.2 | 91.4 | — | — | 98.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-B, FourierModel Type=Ablations2024.08 | 89.26 | — | — | — | 87.42 | — | — | — | — | — | — | — | — | 88.55 | 95.03 | 62.4 | 91.44 | — | — | 97.82 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-B, Fourier+SparseModel Type=Ablations2024.08 | 89.26 | — | — | — | 79.56 | — | — | — | — | — | — | — | — | 89.07 | 95.55 | 62.1 | 91.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-BModel Type=Ours2024.08 | 89.26 | — | — | — | 87.81 | — | — | — | — | — | — | — | — | 90.37 | 95.55 | 62.4 | 91.44 | — | — | 97.82 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-B, FourierType=Ablation, Variant=Fourier2024.08 | 89.26 | — | — | — | 87.42 | — | — | — | — | — | — | — | — | 88.55 | 95.03 | 62.4 | 91.44 | — | — | 97.82 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-B, Fourier+SparseType=Ablation, Variant=Fourier+Sparse2024.08 | 89.26 | — | — | — | 79.56 | — | — | — | — | — | — | — | — | 89.07 | 95.55 | 62.1 | 91.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-BType=Ours2024.08 | 89.26 | — | — | — | 87.81 | — | — | — | — | — | — | — | — | 90.37 | 95.55 | 62.4 | 91.44 | — | — | 97.82 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SGConvModel Type=Convolutional Models2024.08 | 89.2 | — | — | — | 87.17 | — | — | — | — | — | — | — | — | 87.97 | 95.46 | 61.45 | 91.11 | — | — | 97.83 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SGConvCategory=Convolutional Models2024.08 | 89.2 | — | — | — | 87.17 | — | — | — | — | — | — | — | — | 87.97 | 95.46 | 61.45 | 91.11 | — | — | 97.83 | — | — | — | — | — | — | — | — | — | — | — | — | |
| FRACTALModel Category=State Space Models2026.05 | 89.1 | — | — | — | 87.11 | — | — | — | — | — | — | — | — | 87.3 | 94.8 | 61.85 | 91.19 | — | — | 98.39 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Liquid-S4Inference Complexity=Linear, Hybrid=false2023.06 | 89.02 | — | — | — | 87.32 | — | — | — | — | — | — | — | — | 89.5 | 94.8 | 62.75 | 91.2 | — | — | 96.66 | 1.2 | 0.17 | — | — | — | — | — | — | — | — | — | — | |
| Liquid-S4Model Type=State Space Models2024.08 | 89.02 | — | — | — | 87.32 | — | — | — | — | — | — | — | — | 89.5 | 94.8 | 62.75 | 91.2 | — | — | 96.66 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Liquid-S4*Category=State Space Models2024.08 | 89.02 | — | — | — | 87.32 | — | — | — | — | — | — | — | — | 89.5 | 94.8 | 62.75 | 91.2 | — | — | 96.66 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Liquid-S42024.07 | 89 | — | — | — | 87.3 | — | — | — | — | — | — | — | — | 89.5 | 94.8 | 62.8 | 91.2 | — | — | 96.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer + SPTModel Type=Quadratic-Time Transformers2024.08 | 88.81 | — | — | — | 81.81 | — | — | — | — | — | — | — | — | 76 | 88.49 | 59.15 | 90.38 | — | — | 88.05 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer + SPTModel Type=Transformer + SPT2024.08 | 88.81 | — | — | — | 81.81 | — | — | — | — | — | — | — | — | 76 | 88.49 | 59.15 | 90.38 | — | — | 88.05 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S5Model Category=State Space Models2026.05 | 88.72 | — | — | — | 87.04 | — | — | — | — | — | — | — | — | 87.59 | 95.04 | 61.1 | 91.27 | — | — | 98.62 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-L, DilatedModel Type=Ablations2024.08 | 88.36 | — | — | — | 79.19 | — | — | — | — | — | — | — | — | 90.55 | 95.22 | 61.25 | 89.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-L, DilatedType=Ablation, Variant=Dilated2024.08 | 88.36 | — | — | — | 79.19 | — | — | — | — | — | — | — | — | 90.55 | 95.22 | 61.25 | 89.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| H3layer_type=bidirectional2022.12 | 88.2 | — | — | — | 84.8 | — | — | — | — | — | — | — | — | 87.3 | 93 | 57.5 | 91 | — | — | 91.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| H3Inference Complexity=Linear, Hybrid=true2023.06 | 88.2 | — | — | — | 84.8 | — | — | — | — | — | — | — | — | 87.3 | 93 | 57.5 | 91 | — | — | 91.8 | 6 | 0.24 | — | — | — | — | — | — | — | — | — | — | |
| S4DEvaluation=mean of 2 runs2023.11 | 88.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.3 | 91.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| -S4DDescription=Ablation of RIR-EBT-GRC removing S4D layer, Evaluation=mean of 2 runs2023.11 | 88.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.53 | 89.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RIR-EBT-GRCEvaluation=mean of 2 runs2023.11 | 87.97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.04 | 88.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BSTModel Type=Linear-Time Transformers2024.08 | 87.63 | — | — | — | 86.96 | — | — | — | — | — | — | — | — | 91.07 | 95.75 | 61.49 | 90.51 | — | — | 95.28 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BSTCategory=Linear-Time Transformers2024.08 | 87.63 | — | — | — | 86.96 | — | — | — | — | — | — | — | — | 91.07 | 95.75 | 61.49 | 90.51 | — | — | 95.28 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4DSource=copied from [30]2023.11 | 87.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60.18 | 91.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4D2024.10 | 87.34 | — | — | — | — | — | — | — | — | — | — | — | — | 88.19 | 93.96 | 60.52 | 91.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4Dlayer_type=bidirectional2022.12 | 87.3 | — | — | — | 85 | — | — | — | — | — | — | — | — | 87.5 | 93.6 | 58.3 | 90.7 | — | — | 92.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| RIR-GRCEvaluation=mean of 2 runs2023.11 | 86.89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 20.18 | 88.24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GRIL2024.10 | 86.86 | — | — | — | — | — | — | — | — | — | — | — | — | 72.56 | 72.86 | 62.4 | 87.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GRIL2024.10 | 86.86 | — | — | — | — | — | — | — | — | — | — | — | — | 72.56 | 72.86 | 62.4 | 87.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S42021.10 | 86.82 | — | — | — | 86.09 | — | — | — | — | — | — | — | — | 88.65 | 94.2 | 59.6 | 90.9 | 96.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4-v22022.09 | 86.82 | — | — | — | 86.09 | — | — | — | — | — | — | — | — | 88.65 | 94.2 | 59.6 | 90.9 | 96.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S42023.05 | 86.82 | — | — | — | 86.09 | — | — | — | — | — | — | — | — | 88.65 | 94.2 | 59.6 | 90.9 | — | — | 96.35 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4-LegSModel Type=State Space Models2024.08 | 86.82 | — | — | — | 86.09 | — | — | — | — | — | — | — | — | 88.65 | 94.2 | 59.6 | 90.9 | — | — | 96.35 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4-LegSCategory=State Space Models2024.08 | 86.82 | — | — | — | 86.09 | — | — | — | — | — | — | — | — | 88.65 | 94.2 | 59.6 | 90.9 | — | — | 96.35 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4Model Category=State Space Models2026.05 | 86.82 | — | — | — | 86.09 | — | — | — | — | — | — | — | — | 88.65 | 94.2 | 59.6 | 90.9 | — | — | 96.35 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S42024.10 | 86.82 | — | — | — | — | — | — | — | — | — | — | — | — | 88.65 | 94.2 | 59.6 | 90.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4Implementation Source=Paper results2023.03 | 86.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | 94.2 | 59.6 | 90.9 | — | 91.1 | 96.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S42024.07 | 86.8 | — | — | — | 86.5 | — | — | — | — | — | — | — | — | 91.1 | 94.2 | 59.6 | 90.9 | — | — | 96.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4-v2replicated=true, Speed=4.8x, Mem.=0.14x2022.09 | 86.53 | — | — | — | 85.86 | — | — | — | — | — | — | — | — | 88.48 | 94.01 | 59.1 | 90.94 | 96.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4Inference Complexity=Linear, Hybrid=false2023.06 | 86.53 | — | — | — | 85.86 | — | — | — | — | — | — | — | — | 88.48 | 94.01 | 59.1 | 90.94 | — | — | 96.07 | 4.8 | 0.14 | — | — | — | — | — | — | — | — | — | — | |
| S4DImplementation Source=Our reproduction2023.03 | 86.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | 94.2 | 60.2 | 89.5 | — | 91.5 | 97.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-B, DilatedModel Type=Ablations2024.08 | 86.38 | — | — | — | 78.4 | — | — | — | — | — | — | — | — | 90.37 | 94.42 | 60.9 | 88.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MRConv-B, DilatedType=Ablation, Variant=Dilated2024.08 | 86.38 | — | — | — | 78.4 | — | — | — | — | — | — | — | — | 90.37 | 94.42 | 60.9 | 88.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S5Implementation Source=Our reproduction2023.03 | 86.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | 95.7 | 58.5 | 88.9 | — | 88.8 | 96 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4D-LegSImplementation Source=Paper results2023.03 | 86.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | 93.1 | 60.5 | 89.5 | — | 89.9 | 91.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4D2024.07 | 86.2 | — | — | — | 84.9 | — | — | — | — | — | — | — | — | 88.2 | 93.1 | 60.5 | 89.5 | — | — | 92 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4D-LegSInference Complexity=Linear, Hybrid=false2023.06 | 86.18 | — | — | — | 84.89 | — | — | — | — | — | — | — | — | 88.19 | 93.06 | 60.47 | 89.46 | — | — | 91.95 | 6.1 | 0.14 | — | — | — | — | — | — | — | — | — | — | |
| S4D-LegSModel Type=State Space Models2024.08 | 86.18 | — | — | — | 84.89 | — | — | — | — | — | — | — | — | 88.19 | 93.06 | 60.47 | 89.46 | — | — | 91.95 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4D-LegSCategory=State Space Models2024.08 | 86.18 | — | — | — | 84.89 | — | — | — | — | — | — | — | — | 88.19 | 93.06 | 60.47 | 89.46 | — | — | 91.95 | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4DModel Category=State Space Models2026.05 | 86.18 | — | — | — | 84.89 | — | — | — | — | — | — | — | — | 88.19 | 93.06 | 60.47 | 89.46 | — | — | 91.95 | — | — | — | — | — | — | — | — | — | — | — | — | |
| RWKV2023.05 | 86.04 | — | — | — | 72.07 | — | — | — | — | — | — | — | — | 70.53 | 58.42 | 55.88 | 88.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BBT-GRCEvaluation=mean of 2 runs2023.11 | 85.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 55.15 | 90.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSSModel Category=State Space Models2026.05 | 84.8 | — | — | — | 80.73 | — | — | — | — | — | — | — | — | 84.4 | 85 | 57.6 | 87.6 | — | — | 85 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CCNNModel Type=Convolutional Models2024.08 | 84.08 | — | — | — | — | — | — | — | — | — | — | — | — | 88.9 | 91.51 | 43.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CCNNCategory=Convolutional Models2024.08 | 84.08 | — | — | — | — | — | — | — | — | — | — | — | — | 88.9 | 91.51 | 43.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mamba (S6)2024.10 | 81.34 | — | — | — | — | — | — | — | — | — | — | — | — | 65.08 | 69.26 | 38.02 | 80.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mamba (S6)2024.10 | 81.34 | — | — | — | — | — | — | — | — | — | — | — | — | 65.08 | 69.26 | 38.02 | 80.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| S4-v12022.09 | 76.02 | — | — | — | 80.48 | — | — | — | — | — | — | — | — | 87.26 | 86.05 | 58.35 | 87.09 | 88.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RG-LRU2024.10 | 71.75 | — | — | — | — | — | — | — | — | — | — | — | — | 61.15 | 73.38 | 32.34 | 66.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Polaformer2026.06 | 69.65 | — | — | — | 57.23 | — | — | — | — | — | — | — | — | 40.95 | 73.26 | 36.93 | 65.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flexformer_svariant=stationary2026.06 | 67.38 | — | — | — | 59.11 | — | — | — | — | — | — | — | — | 43.39 | 76.15 | 37.09 | 71.55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Linear Trans.Result Source=Tay et al. (2021a), Hardware=TPU2021.05 | 65.9 | — | — | — | 50.55 | — | — | — | — | — | — | — | — | 42.34 | 75.3 | 16.13 | 53.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Linear Trans.2021.10 | 65.9 | — | — | — | 50.46 | — | — | — | — | — | — | — | — | 42.34 | 75.3 | 16.13 | 53.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Linear Transformer2023.05 | 65.9 | — | — | — | 50.46 | — | — | — | — | — | — | — | — | 42.34 | 75.3 | 16.13 | 53.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LTSegments used=12026.01 | 65.9 | — | — | — | 50.5 | — | — | — | — | — | — | — | — | 42.3 | 75.3 | 16.1 | 53.1 | — | — | — | — | — | — | — | — | — | — | 4.7 | 5.7 | 3.9 | 6.2 | 6.2 |