Question Answering on NQ (Accuracy)
70AccuracyOSCAR-llama
Evaluation results
| Method | Links | |
|---|---|---|
| OSCAR-llamaBackbone=Mistral-24B2025.03 | 70 | |
| ProvenceBackbone=Mistral-7B2025.03 | 69 | |
| No compressionBackbone=Mistral-7B2025.03 | 68 | |
| OSCAR-llamaBackbone=Mistral-7B2025.03 | 68 | |
| No compressionBackbone=Mistral-24B2025.03 | 68 | |
| RECOMPBackbone=Mistral-7B2025.03 | 67 | |
| OSCAR-8-LayersBackbone=Mistral-7B2025.03 | 67 | |
| Vanilla RAGModel=GPT-4o-mini, Attack Scenario=Benign, k=102025.09 | 66.6 | |
| MISModel=GPT-4o-mini, Attack Scenario=Benign, k=102025.09 | 66 | |
| OSCAR-5-LayersBackbone=Mistral-7B2025.03 | 66 | |
| OSCAR-llamaBackbone=Qwen-7B2025.03 | 66 | |
| MISModel=GPT-4o-mini, Attack Scenario=@Pos 10, k=102025.09 | 65.4 | |
| PISCOBackbone=Mistral-7B2025.03 | 65 | |
| InstructRAGModel=Mistral-7B, Attack Scenario=Benign, k=102025.09 | 64 | |
| No compressionBackbone=Qwen-7B2025.03 | 64 | |
| OSCAR-8-LayersBackbone=Qwen-7B2025.03 | 64 | |
| AstuteRAGModel=Llama3.2-3B, Attack Scenario=Benign, k=102025.09 | 62.2 | |
| RobustRAGModel=GPT-4o-mini, Attack Scenario=Benign, k=102025.09 | 60.4 | |
| InstructRAGModel=Llama3.2-3B, Attack Scenario=Benign, k=102025.09 | 60.2 | |
| MISModel=Llama3.2-3B, Attack Scenario=Benign, k=102025.09 | 60.2 | |
| MISModel=Mistral-7B, Attack Scenario=Benign, k=102025.09 | 60 | |
| MISModel=GPT-4o-mini, Attack Scenario=@Pos 1, k=102025.09 | 59.6 | |
| RobustRAGModel=GPT-4o-mini, Attack Scenario=@Pos 10, k=102025.09 | 59.4 | |
| MISModel=Llama3.2-3B, Attack Scenario=@Pos 10, k=102025.09 | 59 | |
| AstuteRAGModel=GPT-4o-mini, Attack Scenario=Benign, k=102025.09 | 59 | |
| OSCAR-5-LayersBackbone=Llama-1B2025.03 | 59 | |
| Vanilla RAGModel=Llama3.2-3B, Attack Scenario=Benign, k=102025.09 | 58.4 | |
| MISModel=Mistral-7B, Attack Scenario=@Pos 10, k=102025.09 | 58 | |
| AstuteRAGModel=GPT-4o-mini, Attack Scenario=@Pos 1, k=102025.09 | 58 | |
| RobustRAGModel=GPT-4o-mini, Attack Scenario=@Pos 1, k=102025.09 | 57.6 | |
| MISModel=Llama3.2-3B, Attack Scenario=@Pos 1, k=102025.09 | 57 | |
| Vanilla RAGModel=Mistral-7B, Attack Scenario=Benign, k=102025.09 | 56.2 | |
| AstuteRAGModel=Mistral-7B, Attack Scenario=Benign, k=102025.09 | 56.2 | |
| AstuteRAGModel=GPT-4o-mini, Attack Scenario=@Pos 10, k=102025.09 | 55.4 | |
| MISModel=Mistral-7B, Attack Scenario=@Pos 1, k=102025.09 | 54.8 | |
| InstructRAGModel=GPT-4o-mini, Attack Scenario=Benign, k=102025.09 | 54.8 | |
| No compressionBackbone=Llama-1B2025.03 | 54 | |
| ASTUTE RAGLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 52.2 | |
| RobustRAGModel=Llama3.2-3B, Attack Scenario=@Pos 10, k=102025.09 | 52.2 | |
| InstructRAGModel=Mistral-7B, Attack Scenario=@Pos 1, k=102025.09 | 51.4 | |
| RobustRAGModel=Llama3.2-3B, Attack Scenario=Benign, k=102025.09 | 51.4 | |
| USCLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 51.2 | |
| RobustRAGModel=Llama3.2-3B, Attack Scenario=@Pos 1, k=102025.09 | 50.4 | |
| ASTUTE RAGLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 50.2 | |
| ASTUTE RAGLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 50.2 | |
| RankCoTBackbone=Qwen2.5-14B-Instruct2025.02 | 49.98 | |
| AstuteRAGModel=Mistral-7B, Attack Scenario=@Pos 1, k=102025.09 | 49.8 | |
| InstructRAGModel=GPT-4o-mini, Attack Scenario=@Pos 1, k=102025.09 | 49.4 | |
| RankCoTBackbone=MiniCPM3-4B2025.02 | 48.78 | |
| USCLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 48.1 | |
| RobustRAGLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 47.8 | |
| No RefinementBackbone=Qwen2.5-14B-Instruct2025.02 | 47.66 | |
| Self-RouteLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 47.5 | |
| Self-RouteLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 47.5 | |
| RankCoTBackbone=Llama3-8B-Instruct2025.02 | 47.41 | |
| No RAGLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 47.1 | |
| InstructRAGLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 47.1 | |
| InstructRAGLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 46.8 | |
| No RAGLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 46.8 | |
| USCLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 46.4 | |
| RobustRAGModel=Mistral-7B, Attack Scenario=Benign, k=102025.09 | 46.4 | |
| RerankBackbone=Llama3-8B-Instruct2025.02 | 46.18 | |
| No RAGBackbone=Mistral-7B2025.03 | 46 | |
| CDKCBackbone=Qwen2.5-14B-Instruct2026.02 | 45.7 | |
| No RefinementBackbone=Llama3-8B-Instruct2025.02 | 45.68 | |
| InstructRAGLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 45.4 | |
| Self-RouteLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 45.4 | |
| CoTBackbone=Llama3-8B-Instruct2025.02 | 45.33 | |
| GRPOBackbone=Qwen2.5-14B-Instruct2026.02 | 45.27 | |
| GenReadLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 45.1 | |
| No RAGLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 44.8 | |
| RAG (Open-domain)Model setting=Finetuned2023.02 | 44.5 | |
| RAGLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 44.4 | |
| RobustRAGModel=Mistral-7B, Attack Scenario=@Pos 1, k=102025.09 | 44.4 | |
| SummaryBackbone=Llama3-8B-Instruct2025.02 | 44.27 | |
| RobustRAGModel=Mistral-7B, Attack Scenario=@Pos 10, k=102025.09 | 44 | |
| RAGLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 43.1 | |
| RAGLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 42.7 | |
| RobustRAGLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 42.7 | |
| ASTUTE RAGLLM=Mistral-Nemo (2407), 12B, Evaluation Setting=zero-shot2024.10 | 42.7 | |
| No RefinementBackbone=MiniCPM3-4B2025.02 | 42.51 | |
| GenReadLLM=Claude 3.5 Sonnet (20240620), Evaluation Setting=zero-shot2024.10 | 42 | |
| Self-RouteLLM=Mistral-Nemo (2407), 12B, Evaluation Setting=zero-shot2024.10 | 41.4 | |
| Vanilla RAGModel=GPT-4o-mini, Attack Scenario=@Pos 10, k=102025.09 | 41 | |
| GenReadLLM=Mistral-Large (2407), 128B, Evaluation Setting=zero-shot2024.10 | 40.7 | |
| Vanilla RAGModel=Mistral-7B, Attack Scenario=@Pos 1, k=102025.09 | 40 | |
| RAGLLM=Mistral-Nemo (2407), 12B, Evaluation Setting=zero-shot2024.10 | 39.3 | |
| RAGBackbone=Qwen2.5-14B-Instruct2026.02 | 38.97 | |
| InstructRAGModel=GPT-4o-mini, Attack Scenario=@Pos 10, k=102025.09 | 38.8 | |
| GenReadLLM=Mistral-Nemo (2407), 12B, Evaluation Setting=zero-shot2024.10 | 38.6 | |
| InstructRAGLLM=Mistral-Nemo (2407), 12B, Evaluation Setting=zero-shot2024.10 | 38.3 | |
| Vanilla RAGModel=Llama3.2-3B, Attack Scenario=@Pos 1, k=102025.09 | 37.4 | |
| AstuteRAGModel=Mistral-7B, Attack Scenario=@Pos 10, k=102025.09 | 36.4 | |
| RobustRAGLLM=Mistral-Nemo (2407), 12B, Evaluation Setting=zero-shot2024.10 | 35.6 | |
| RobustRAGLLM=Gemini 1.5 Pro (002), Evaluation Setting=zero-shot2024.10 | 34.2 | |
| RAGBackbone=Qwen2.5-3B-Instruct2026.02 | 33.37 | |
| Vanilla RAGModel=GPT-4o-mini, Attack Scenario=@Pos 1, k=102025.09 | 31.2 | |
| CGKEBackbone=Qwen2.5-14B-Instruct2026.02 | 30.6 | |
| CDKCBackbone=Qwen2.5-3B-Instruct2026.02 | 30.33 | |
| CoTBackbone=Qwen2.5-14B-Instruct2026.02 | 29.83 |