Sequence Classification on ListOps
98.85Accuracy (%)Transformer (Full Attn)
Evaluation Results
| Method | Links | |
|---|---|---|
| Transformer (Full Attn)Source=This work, Tokenization=GPT-2 BPE2026.06 | 98.85 | |
| CHIAR DCT+AttnSource=This work, Tokenization=GPT-2 BPE2026.06 | 63.35 | |
| CMRUDepth (r)=6, State dimension (d)=322026.05 | 61.39 | |
| αCMRUDepth (r)=6, State dimension (d)=322026.05 | 61.36 | |
| αCMRUDepth (r)=3, State dimension (d)=322026.05 | 60.86 | |
| CMRUDepth (r)=3, State dimension (d)=322026.05 | 60.42 | |
| MINGRUDepth (r)=6, State dimension (d)=322026.05 | 59.69 | |
| HKT-Larged=512, Overhead=1.313×2026.04 | 55.6 | |
| HKT-Mediumd=256, Overhead=1.313×2026.04 | 55.5 | |
| HKT-Smalld=128, Overhead=1.313×2026.04 | 55.1 | |
| MINGRUDepth (r)=3, State dimension (d)=322026.05 | 55.07 | |
| MHA (retrained)Overhead=1.00×2026.04 | 50.33 | |
| TransEvolve-randomFF-1hidden size=256, configuration=randomFF-12021.09 | 43.2 | |
| TransEvolve-fullFF-1hidden size=256, configuration=fullFF-12021.09 | 42.2 | |
| LRUDepth (r)=3, State dimension (d)=322026.05 | 40.45 | |
| MINGRUDepth (r)=1, State dimension (d)=322026.05 | 39.4 | |
| LRUDepth (r)=6, State dimension (d)=322026.05 | 39.37 | |
| TransEvolve-randomFF-2hidden size=256, configuration=randomFF-22021.09 | 39.1 | |
| CMRUDepth (r)=1, State dimension (d)=322026.05 | 38.84 | |
| αCMRUDepth (r)=1, State dimension (d)=322026.05 | 38.74 | |
| LRUDepth (r)=1, State dimension (d)=322026.05 | 38.64 | |
| TransEvolve-fullFF-2hidden size=256, configuration=fullFF-22021.09 | 37.8 | |
| Reformer2021.09 | 37.3 | |
| Synthesizer2021.09 | 37 | |
| Random Feature Attention (RFA)2021.09 | 36.8 | |
| Transformerhidden size=5122021.09 | 36.4 | |
| TransformerSource=Tay et al. 2021, Tokenization=Character-level2026.06 | 36.37 | |
| Big Bird2021.09 | 36 | |
| Linformer2021.09 | 35.7 | |
| LinformerSource=Tay et al. 2021, Tokenization=Character-level2026.06 | 35.7 | |
| FNetSource=Tay et al. 2021, Tokenization=Character-level2026.06 | 35.33 | |
| PerformerSource=Tay et al. 2021, Tokenization=Character-level2026.06 | 18.01 | |
| Performers2021.09 | 18 | |
| Sinkhorn2021.09 | 17.1 | |
| Linear attention2021.09 | 16.1 |