Long-range Dependency Modeling and Efficiency on Long Range Arena (LRA)
66.69Text (4K) AccShiftAdd-Transformer
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| ShiftAdd-TransformerBatch Size=1, Hardware Accelerator=Eyeriss-like accelerator2023.06 | 66.69 | 37.15 | 82.02 | 35.62 | 55.37 | 7.38 | 8.53 | |
| TransformerBatch Size=1, Hardware Accelerator=Eyeriss-like accelerator2023.06 | 65.02 | 37.1 | 79.35 | 38.2 | 54.92 | 84.54 | 139.83 | |
| ReformerBatch Size=1, Hardware Accelerator=Eyeriss-like accelerator2023.06 | 64.88 | 19.05 | 78.64 | 43.29 | 51.47 | 11.19 | 19.04 | |
| PerformerBatch Size=1, Hardware Accelerator=Eyeriss-like accelerator2023.06 | 63.81 | 18.8 | 78.62 | 37.07 | 49.58 | 11.93 | 18.74 | |
| LinformerBatch Size=1, Hardware Accelerator=Eyeriss-like accelerator2023.06 | 55.91 | 37.25 | 79.37 | 37.84 | 52.59 | 12.13 | 19.68 |