Retrieval-Augmented Generation on ICR2

87NQ ScorePhi-3-7B-128K

Evaluation results

Method
Phi-3-7B-128K2025.01	87	81	91	14	68
Qwen-2-7B-32K2025.01	83	80	91	17	68
Mistral-2-7B-32K2025.01	83	81	94	18	69
Oracle RAG2025.01	83	81	94	18	—
LLaMA-3-instruct-8B2025.01	80	74	86	19	65
GPT-4-turbo2025.01	79	82	94	18	68
GPT-4 Oracle2025.01	79	82	94	18	—
Qwen-2-1.5B-32K2025.01	77	65	86	15	61
GPT-4-turbo2025.01	67	64	92	15	59
GPT-4 Vanilla RAG2025.01	67	64	92	15	—
SFT-DA w/ RAP2025.01	64	67	89	21	—
SFT-Cite-Context-ID2025.01	63	63	89	21	—
SFT-RTA w/ RAP2025.01	63	71	92	23	—
Mistral-7B Re-ranking (TinyBERT)2025.01	62	51	91	13	—
GPT-4-turbo2025.01	61	50	85	16	53
GPT-4 Close-book2025.01	61	50	85	16	—
Mistral-7B Re-ranking (MiniLM)2025.01	61	47	92	12	—
Phi-3-7B-128K2025.01	60	50	90	14	53
SFT-Retrieve-then-Answer2025.01	60	67	91	22	—
SFT-Direct Answer2025.01	59	70	90	22	—
Qwen-2-7B-32K2025.01	57	47	88	13	51
Mistral-7B RAG w/ RTA Prompting2025.01	54	51	83	15	—
Mistral-2-7B-32K2025.01	53	50	81	13	49
LLaMA-3-instruct-8B2025.01	53	45	83	15	49
Mistral-7B Vanilla RAG2025.01	53	50	81	13	—
Mistral-7B Re-ranking (TinyBERT)2025.01	52	47	88	12	—
Mistral-7B Re-ranking (MiniLM)2025.01	51	46	86	13	—
Mistral-2-7B-32K2025.01	49	38	73	14	43
Mistral-7B Close-book2025.01	49	38	73	14	—
RetHead w/ Lgen + Lret2025.01	48	54	90	21	—
Phi-3-7B-128K2025.01	46	32	73	13	41
LLaMA-3-instruct-8B2025.01	44	19	53	14	32
RetHead w/ Lret2025.01	39	44	82	13	—
Qwen-2-7B-32K2025.01	35	30	71	15	38
Qwen-2-1.5B-32K2025.01	35	32	78	12	39
Qwen-2-1.5B-32K2025.01	29	24	62	15	32
RetHead w/ Lgen2025.01	28	25	82	18	—