Recall task on Content-dependent recall L=17 (test)
83.8AccuracyNonlinear MLP-Mixer
Evaluation results
| Method | Links | |
|---|---|---|
| Nonlinear MLP-Mixerparams=3,770, sequence length (L)=172026.06 | 83.8 | |
| Attentionparams=5,594, sequence length (L)=172026.06 | 76 | |
| FlowYB (ours, integrable)params=1,970, sequence length (L)=172026.06 | 75.9 | |
| Orthogonal RNNparams=2,570, sequence length (L)=172026.06 | 75.5 | |
| Diagonal SSM (S4D-like)params=1,466, sequence length (L)=172026.06 | 65.6 | |
| No-mixing (control)params=1,394, sequence length (L)=172026.06 | 49.8 |