Scientific document Question Answering on Qasper
11.5Exact Match (EM)Dense
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Densek=5, Generator=Meta-Llama-3-8B-Instruct2026.06 | 11.5 | 26.8 | 168 | |
| Qwen3-Emb-0.6Bk=2, Generator=Meta-Llama-3-8B-Instruct2026.06 | 10 | 26.6 | 80 | |
| BM25k=5, Generator=Meta-Llama-3-8B-Instruct2026.06 | 9.5 | 23.6 | 187 | |
| Densek=2, Generator=Meta-Llama-3-8B-Instruct2026.06 | 8.5 | 21.1 | 76 | |
| Qwen3-Emb-0.6Bk=1, Generator=Meta-Llama-3-8B-Instruct2026.06 | 8.5 | 19.8 | 47 | |
| Qwen3-Emb-0.6Bk=5, Generator=Meta-Llama-3-8B-Instruct2026.06 | 8.5 | 30.2 | 171 | |
| BM25k=2, Generator=Meta-Llama-3-8B-Instruct2026.06 | 8 | 20.5 | 82 | |
| BM25k=1, Generator=Meta-Llama-3-8B-Instruct2026.06 | 6 | 17.1 | 50 | |
| Latent Memoryk=5, Memory tokens=1, Generator=Meta-Llama-3-8B-Instruct2026.06 | 6 | 12.3 | 61 | |
| Densek=1, Generator=Meta-Llama-3-8B-Instruct2026.06 | 5.5 | 16.6 | 46 | |
| Latent Memoryk=1, Memory tokens=8, Generator=Meta-Llama-3-8B-Instruct2026.06 | 5 | 14.5 | 32 | |
| Latent Memoryk=2, Memory tokens=8, Generator=Meta-Llama-3-8B-Instruct2026.06 | 5 | 14.3 | 48 | |
| Latent Memoryk=5, Memory tokens=8, Generator=Meta-Llama-3-8B-Instruct2026.06 | 5 | 13.2 | 96 | |
| Latent Memoryk=2, Memory tokens=1, Generator=Meta-Llama-3-8B-Instruct2026.06 | 4.5 | 12.2 | 34 | |
| Latent Memoryk=1, Memory tokens=1, Generator=Meta-Llama-3-8B-Instruct2026.06 | 3 | 10.9 | 25 | |
| Full ContextGenerator=Meta-Llama-3-8B-Instruct2026.06 | 2.5 | 16.5 | 4,701 |