Long Dependency Question Answering on LooGLE
40RetrievalLongLLMLingua
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| LongLLMLinguaTokens=3,121, 1/T=10x2023.10 | 40 | 35 | 19.7 | 33.6 | 32.1 | — | — | — | |
| LongLLMLingua w/o ReorderTokens=3,119, 1/T=10x2023.10 | 39.3 | 33.8 | 18.7 | 31.6 | 30.9 | — | — | — | |
| LongLLMLingua rkTokens=3,158, 1/T=10x2023.10 | 38.6 | 32.2 | 16.2 | 26.3 | 28.3 | — | — | — | |
| LIFTModel=Qwen 32025.02 | 35.76 | 43.28 | 18.56 | 47.75 | — | — | 55.86 | 40.04 | |
| ICLModel=Qwen 32025.02 | 30.26 | 40.47 | 22 | 44.58 | — | — | 47.36 | 36.78 | |
| SBERTTokens=3,169, 1/T=10x2023.10 | 28.9 | 21.1 | 10.7 | 27.2 | 22 | — | — | — | |
| LIFTModel=Gemma 22025.02 | 24.74 | 59.07 | 12 | 34.98 | — | — | 42.44 | 34.06 | |
| Original PromptTokens=30,5462023.10 | 24.1 | 20.9 | 13.5 | 32.1 | 22.6 | — | — | — | |
| LIFTBackbone=Llama-3-8B-Instruct, Synthetic QA Pairs=102025.02 | 22.63 | 39.53 | 16 | 27.83 | — | — | 52.69 | 27.25 | |
| LIFTBackbone=Llama-3-8B-Instruct, Synthetic QA Pairs=52025.02 | 21.58 | 36.28 | 14 | 29.8 | — | — | 45.67 | 26.79 | |
| ICLModel=Gemma 22025.02 | 21.58 | 40 | 10 | 36.95 | — | — | 37.37 | 29.79 | |
| Finetune-RawBackbone=Llama-3-8B-Instruct2025.02 | 20.53 | 27.44 | 18 | 30.05 | — | — | 33.93 | 25.16 | |
| BM25Tokens=3,185, 1/T=10x2023.10 | 20.4 | 21.7 | 8.2 | 26.3 | 19.2 | — | — | — | |
| LLMLinguaBackbone=Llama-3-8B-Instruct2025.02 | 20 | 19.53 | 8 | 33.5 | — | — | 24.14 | 23.8 | |
| TempLoRABackbone=Llama-3-8B-Instruct2025.02 | 19.47 | 32.56 | 12 | 32.02 | — | — | 29.67 | 25.97 | |
| LlamaIndexBackbone=Llama-3-8B-Instruct2025.02 | 17.11 | 9.77 | 12 | 33 | — | — | 41.93 | 21.07 | |
| Selective-ContextTokens=3,710, 1/T=8x2023.10 | 16.7 | 5 | 2.3 | 17.6 | 10.4 | — | — | — | |
| MemoryLLMBackbone=Llama-3-8B-Instruct2025.02 | 15.53 | 18.14 | 8 | 29.31 | — | — | 33.06 | 20.44 | |
| Truncated ICLBackbone=Llama-3-8B-Instruct2025.02 | 15.26 | 1.86 | 5 | 25.37 | — | — | 44.49 | 15.44 | |
| LLMLinguaTokens=3,404, 1/T=9x2023.10 | 10 | 25 | 13.3 | 21.1 | 17.3 | — | — | — | |
| Zero-shotTokens=43, 1/T=710x2023.10 | 8.7 | 6.3 | 1.2 | 14.5 | 7.7 | — | — | — | |
| Llama3-ChatQA2-8BParameters=8B2025.02 | — | — | — | — | — | 50.7 | — | — | |
| Llama3.1-8B-instParameters=8B2025.02 | — | — | — | — | — | 56.1 | — | — | |
| Llama3.1-OpAmp-8BParameters=8B2025.02 | — | — | — | — | — | 56.6 | — | — | |
| Mistral-7B-inst-v0.3Parameters=7B2025.02 | — | — | — | — | — | 51.6 | — | — | |
| Qwen2.5-7B-instParameters=7B2025.02 | — | — | — | — | — | 53.8 | — | — |