Multi-Paper Question Answering on ScholarQABench Multi
4.32Mean ScoreLlama 3.1 8B Instruct
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Llama 3.1 8B Instructretrieval=without2026.04 | 4.32 | 4.01 | 4.19 | 4.76 | 475 | 0 | 0 | 0 | |
| Llama 3.2 3B Instructretrieval=without2026.04 | 4.19 | 3.96 | 4.06 | 4.54 | 450 | 0 | 0 | 0 | |
| OS-8Bexternal_baseline=true2026.04 | 4.12 | 3.92 | 4.44 | 4.02 | 578.6 | — | — | 42.8 | |
| Llama 3.1 8B Instructretrieval=with2026.04 | 4.04 | 3.83 | 3.9 | 4.38 | 453 | 23.5 | 26.45 | 24.89 | |
| Full FTEpochs=2, Steps=16k2026.04 | 4 | 3.89 | 3.88 | 4.23 | 466 | 21.2 | 19.66 | 20.4 | |
| Full FTEpochs=5, Steps=16k2026.04 | 3.98 | 3.82 | 3.84 | 4.26 | 484 | 23.81 | 23.96 | 23.89 | |
| Full FTEpochs=2, Steps=10k2026.04 | 3.96 | 3.85 | 3.83 | 4.19 | 468 | 24.52 | 22.69 | 23.57 | |
| LoRA FTEpochs=2, Steps=10k, Base Model=Llama 3.1 8B Instruct2026.04 | 3.94 | 3.59 | 3.92 | 4.31 | 1,297 | 25.64 | 22.77 | 24.12 | |
| Full FTEpochs=1, Steps=16k2026.04 | 3.94 | 3.81 | 3.8 | 4.2 | 532 | 24.51 | 23.33 | 23.91 | |
| LoRA FTEpochs=1, Steps=8k, Base Model=Llama 3.1 8B Instruct2026.04 | 3.91 | 3.74 | 3.78 | 4.2 | 784 | 23.41 | 21.06 | 22.18 | |
| LoRA FTEpochs=2, Steps=8k, Base Model=Llama 3.1 8B Instruct2026.04 | 3.88 | 3.65 | 3.81 | 4.18 | 841 | 24.74 | 23.72 | 24.22 | |
| LoRA FTEpochs=5, Steps=16k, Base Model=Llama 3.1 8B Instruct2026.04 | 3.87 | 3.55 | 3.8 | 4.27 | 1,024 | 20.1 | 18.75 | 19.4 | |
| LoRA FTEpochs=2, Steps=16k, Base Model=Llama 3.1 8B Instruct2026.04 | 3.86 | 3.55 | 3.82 | 4.21 | 887 | 24.07 | 25.02 | 24.53 | |
| LoRA FTEpochs=1, Steps=16k, Base Model=Llama 3.1 8B Instruct2026.04 | 3.85 | 3.6 | 3.87 | 4.08 | 1,017 | 23.35 | 21.63 | 22.46 | |
| Llama3-8Bretrieval=without, external_baseline=true2026.04 | 3.79 | — | — | — | — | — | — | 0 | |
| Llama 3.2 3B Instructretrieval=with2026.04 | 3.72 | 3.67 | 3.51 | 3.99 | 363 | 16.49 | 19.05 | 17.68 |