Question Answering on Scrolls NarraQA
12.29AccuracyMistral (Full-Attention)
Evaluation results
| Method | Links | |
|---|---|---|
| Mistral (Full-Attention)Parameters=790M, Zero-shot=true2024.07 | 12.29 | |
| BMoJo (Fading)Parameters=790M, Zero-shot=true2024.07 | 9.66 | |
| Mamba (SSM)Parameters=790M, Zero-shot=true2024.07 | 9.62 | |
| Mistral (Full-Attention)Parameters=370M, Zero-shot=true2024.07 | 9.24 | |
| BMoJo (Fading + Eidetic)Parameters=790M, Zero-shot=true2024.07 | 7.64 | |
| BMoJo (Fading)Parameters=370M, Zero-shot=true2024.07 | 7.48 | |
| Mamba (SSM)Parameters=370M, Zero-shot=true2024.07 | 6.04 | |
| BMoJo (Fading + Eidetic)Parameters=370M, Zero-shot=true2024.07 | 5.92 | |
| Hybrid (Sliding Attention + SSM)Parameters=370M, Zero-shot=true2024.07 | 5.35 | |
| Hybrid (Sliding Attention + SSM)Parameters=790M, Zero-shot=true2024.07 | 3.62 |