Question Answering on HQA
0.442EMLongLLMLin
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| LongLLMLinReader=GPT-5-mini (low)2026.03 | 0.442 | 0.607 | 1.357 | 39.1 | |
| LooCompReader=GPT-5-mini (low)2026.03 | 0.441 | 0.605 | 0.158 | 8.5 | |
| EXITReader=GPT-5-mini (low)2026.03 | 0.433 | 0.59 | 1.474 | 41.9 | |
| ProvenceReader=GPT-5-mini (low)2026.03 | 0.43 | 0.589 | 0.199 | 13.8 | |
| CompActReader=GPT-5-mini (low)2026.03 | 0.418 | 0.571 | 4.696 | 3.7 | |
| LooCompReader=Kimi-K22026.03 | 0.41 | 0.535 | 0.158 | 8.5 | |
| RawReader=Kimi-K22026.03 | 0.408 | 0.534 | — | 100 | |
| RefinerReader=GPT-5-mini (low)2026.03 | 0.399 | 0.548 | 3.915 | 4.4 | |
| LongLLMLinReader=Kimi-K22026.03 | 0.392 | 0.517 | 1.357 | 39.1 | |
| EXITReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 0.386 | 0.502 | 4.7 | — | |
| Original DocsReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-202024.12 | 0.385 | 0.5 | 25.3 | — | |
| RECOMP-absReader=GPT-5-mini (low)2026.03 | 0.385 | 0.531 | 1.206 | 1.8 | |
| LooCompReader=Llama-3.3-70B Instruct, Retrieved chunks=Top-202026.03 | 0.385 | 0.512 | 0.158 | 8.5 | |
| EXITReader=Kimi-K22026.03 | 0.383 | 0.505 | 1.474 | 41.9 | |
| ProvenceReader=Kimi-K22026.03 | 0.383 | 0.501 | 0.199 | 13.8 | |
| RawReader=Gemini-2.5-flash2026.03 | 0.371 | 0.483 | — | 100 | |
| EXITReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 0.37 | 0.483 | 3.3 | — | |
| LongLLMLinguaReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 0.367 | 0.484 | 10.6 | — | |
| LooCompReader=Gemini-2.5-flash2026.03 | 0.365 | 0.475 | 0.158 | 8.5 | |
| CompActReader=Kimi-K22026.03 | 0.361 | 0.468 | 4.696 | 3.7 | |
| RefinerReader=Kimi-K22026.03 | 0.347 | 0.458 | 3.915 | 4.4 | |
| LongLLMLinReader=Gemini-2.5-flash2026.03 | 0.343 | 0.454 | 1.357 | 39.1 | |
| RECOMP-absReader=Kimi-K22026.03 | 0.342 | 0.461 | 1.206 | 1.8 | |
| LongLLMLinguaReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 0.341 | 0.453 | 4 | — | |
| CompActReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 0.338 | 0.441 | 11 | — | |
| RefinerReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 0.338 | 0.447 | 14.6 | — | |
| Original DocsReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-52024.12 | 0.337 | 0.445 | 8.3 | — | |
| EXITReader=Gemini-2.5-flash2026.03 | 0.335 | 0.442 | 1.474 | 41.9 | |
| EXITReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 0.329 | 0.44 | 1.8 | — | |
| RefinerReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 0.319 | 0.422 | 33 | — | |
| ProvenceReader=Gemini-2.5-flash2026.03 | 0.319 | 0.423 | 0.199 | 13.8 | |
| CompActReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 0.317 | 0.409 | 32 | — | |
| RECOMP-AbstReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 0.313 | 0.418 | 4.8 | — | |
| LongLLMLinguaReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 0.312 | 0.414 | 2.8 | — | |
| RECOMP-extReader=Kimi-K22026.03 | 0.312 | 0.42 | 0.035 | 12.4 | |
| LooCompReader=Llama-3.1-8B Instruct, Retrieved chunks=Top-52026.03 | 0.31 | 0.42 | 0.038 | 15.9 | |
| EXITReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 0.306 | 0.415 | 0.8 | — | |
| RECOMP-AbstReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 0.303 | 0.408 | 4.4 | — | |
| CompActReader=Gemini-2.5-flash2026.03 | 0.301 | 0.395 | 4.696 | 3.7 | |
| RECOMP-absReader=Gemini-2.5-flash2026.03 | 0.3 | 0.405 | 1.206 | 1.8 | |
| Original DocsReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-202024.12 | 0.299 | 0.405 | 2.9 | — | |
| RefinerReader=Gemini-2.5-flash2026.03 | 0.293 | 0.386 | 3.915 | 4.4 | |
| CompActReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 0.289 | 0.396 | 27.5 | — | |
| CompActReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 0.288 | 0.398 | 8.3 | — | |
| RefinerReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 0.288 | 0.4 | 6.9 | — | |
| Original DocsReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-52024.12 | 0.281 | 0.386 | 1 | — | |
| LongLLMLinguaReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 0.28 | 0.38 | 0.8 | — | |
| RECOMP-ExtrReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 0.276 | 0.367 | 2.9 | — | |
| RECOMP-AbstReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 0.268 | 0.371 | 2.4 | — | |
| RefinerReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 0.266 | 0.373 | 29.6 | — | |
| RECOMP-AbstReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 0.265 | 0.37 | 2.2 | — | |
| RECOMP-ExtrReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 0.256 | 0.345 | 2.9 | — | |
| RECOMP-ExtrReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 0.234 | 0.328 | 0.4 | — | |
| RECOMP-ExtrReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 0.222 | 0.312 | 0.5 | — | |
| RECOMP-extReader=Gemini-2.5-flash2026.03 | 0.2 | 0.278 | 0.035 | 12.4 |