Induction head on Synthetic sequence modeling Induction head
100AccuracyTransformer
Evaluation Results
| Method | Links | |
|---|---|---|
| TransformerAttention=Softmax2023.12 | 100 | |
| MIMOFormerN=2, variant=att., Attention=FAVOR+2023.12 | 99.4 | |
| MIMOFormerN=2, variant=att., Attention=DPFP2023.12 | 98.56 | |
| PerformerAttention=FAVOR+2023.12 | 31.58 | |
| DataMUXN=2, Attention=Softmax2023.12 | 6.06 |