Question Answering on Scrolls QASPER
14.8AccuracyMistral (Full-Attention)
Evaluation results
| Method | Links | |
|---|---|---|
| Mistral (Full-Attention)Parameters=790M, Zero-shot=true2024.07 | 14.8 | |
| Mistral (Full-Attention)Parameters=370M, Zero-shot=true2024.07 | 13.35 | |
| BMoJo (Fading)Parameters=790M, Zero-shot=true2024.07 | 12.8 | |
| Mamba (SSM)Parameters=790M, Zero-shot=true2024.07 | 12.43 | |
| BMoJo (Fading + Eidetic)Parameters=790M, Zero-shot=true2024.07 | 11.05 | |
| BMoJo (Fading)Parameters=370M, Zero-shot=true2024.07 | 10.29 | |
| Mamba (SSM)Parameters=370M, Zero-shot=true2024.07 | 9.54 | |
| BMoJo (Fading + Eidetic)Parameters=370M, Zero-shot=true2024.07 | 9.02 | |
| Hybrid (Sliding Attention + SSM)Parameters=370M, Zero-shot=true2024.07 | 7.77 | |
| Hybrid (Sliding Attention + SSM)Parameters=790M, Zero-shot=true2024.07 | 7.63 |