Retrieval-Augmented Generation on LOFT
100NQ ScoreMiniLM
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MiniLMK=82025.01 | 100 | 98 | 82 | — | |
| MiniLMK=322025.01 | 100 | 100 | 90 | — | |
| MiniLMK=502025.01 | 100 | 100 | 92 | — | |
| TinyBERTK=82025.01 | 99 | 98 | 80 | — | |
| TinyBERTK=322025.01 | 99 | 99 | 81 | — | |
| TinyBERTK=502025.01 | 99 | 99 | 82 | — | |
| SFT-RTA w/ RAP2025.01 | 95 | 99 | 100 | — | |
| Phi-3-7B-128KRAG Setting=Oracle RAG2025.01 | 90 | 85 | 63 | 79 | |
| Mistral-2-7B-32KRAG Setting=Oracle RAG2025.01 | 89 | 81 | 41 | 70 | |
| Oracle RAGBackbone=Mistral-7B-Instruct, Strategy=Oracle2025.01 | 89 | 81 | 41 | — | |
| Qwen-2-7B-32KRAG Setting=Oracle RAG2025.01 | 88 | 81 | 56 | 75 | |
| GPT-4-turboRAG Setting=Oracle RAG2025.01 | 88 | 87 | 72 | 82 | |
| GPT-4 OracleBackbone=GPT-4, Strategy=Oracle2025.01 | 88 | 87 | 72 | — | |
| Mistral-7B Re-ranking (TinyBERT)Backbone=Mistral-7B-Instruct, Re-ranker=TinyBERT, k=82025.01 | 88 | 78 | 29 | — | |
| Mistral-7B Re-ranking (TinyBERT)Backbone=Mistral-7B-Instruct, Re-ranker=TinyBERT, k=322025.01 | 87 | 87 | 33 | — | |
| GPT-4-turboRAG Setting=Vanilla RAG2025.01 | 85 | 78 | 51 | 71 | |
| GPT-4 Vanilla RAGBackbone=GPT-4, Strategy=Vanilla RAG2025.01 | 85 | 78 | 51 | — | |
| SFT-RTA w/ RAPBackbone=Mistral-7B-Instruct, Decoding Protocol=Retrieval-Attention Probing2025.01 | 85 | 79 | 39 | — | |
| Mistral-7B Re-ranking (MiniLM)Backbone=Mistral-7B-Instruct, Re-ranker=MiniLM, k=322025.01 | 84 | 84 | 39 | — | |
| Mistral-7B Re-ranking (MiniLM)Backbone=Mistral-7B-Instruct, Re-ranker=MiniLM, k=82025.01 | 83 | 77 | 27 | — | |
| LLaMA-3-instruct-8BRAG Setting=Oracle RAG2025.01 | 81 | 77 | 60 | 73 | |
| Qwen-2-1.5B-32KRAG Setting=Oracle RAG2025.01 | 80 | 76 | 44 | 67 | |
| Qwen-2-7B-32KRAG Setting=Vanilla RAG2025.01 | 79 | 61 | 29 | 56 | |
| SFT-DA w/ RAPBackbone=Mistral-7B-Instruct, Decoding Protocol=Retrieval-Attention Probing2025.01 | 78 | 76 | 47 | — | |
| Phi-3-7B-128KRAG Setting=Vanilla RAG2025.01 | 76 | 68 | 41 | 62 | |
| SFT-Cite-Context-IDBackbone=Mistral-7B-Instruct, Strategy=SFT-CCI2025.01 | 76 | 54 | 35 | — | |
| SFT-Retrieve-then-AnswerBackbone=Mistral-7B-Instruct, Strategy=SFT-RTA2025.01 | 74 | 69 | 33 | — | |
| SFT-Direct AnswerBackbone=Mistral-7B-Instruct, Strategy=SFT-DA2025.01 | 70 | 65 | 25 | — | |
| Mistral-2-7B-32KRAG Setting=Vanilla RAG2025.01 | 64 | 62 | 27 | 51 | |
| Mistral-7B Vanilla RAGBackbone=Mistral-7B-Instruct, Strategy=Vanilla RAG2025.01 | 64 | 62 | 27 | — | |
| Qwen-2-1.5B-32KRAG Setting=Vanilla RAG2025.01 | 61 | 43 | 12 | 39 | |
| Mistral-7B RAG w/ RTA PromptingBackbone=Mistral-7B-Instruct, Strategy=RTA Prompting2025.01 | 60 | 70 | 27 | — | |
| GPT-4-turboRAG Setting=Closed-book2025.01 | 58 | 57 | 28 | 48 | |
| GPT-4 Close-bookBackbone=GPT-4, Strategy=Close-book2025.01 | 58 | 57 | 28 | — | |
| Mistral-2-7B-32KRAG Setting=Closed-book2025.01 | 46 | 38 | 5 | 30 | |
| Mistral-7B Close-bookBackbone=Mistral-7B-Instruct, Strategy=Close-book2025.01 | 46 | 38 | 5 | — | |
| LLaMA-3-instruct-8BRAG Setting=Vanilla RAG2025.01 | 45 | 56 | 16 | 39 | |
| LLaMA-3-instruct-8BRAG Setting=Closed-book2025.01 | 41 | 19 | 1 | 20 | |
| Phi-3-7B-128KRAG Setting=Closed-book2025.01 | 39 | 35 | 9 | 28 | |
| Qwen-2-7B-32KRAG Setting=Closed-book2025.01 | 31 | 30 | 7 | 23 | |
| Qwen-2-1.5B-32KRAG Setting=Closed-book2025.01 | 21 | 22 | 3 | 15 | |
| RetHead w/ Lgen + LretBackbone=Mistral-7B-Instruct, Training Objective=Lgen + Lret2025.01 | 15 | 13 | 7 | — |