Contextual Robustness Question Answering on ConflictQA Unknown queries
99.28Accuracy (Short Context)COT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| COTBackbone=Llama-32025.02 | 99.28 | 96.28 | |
| COTIntervention Layer=7-th layer, Evaluation Protocol=Chain-of-Thought2025.02 | 98.32 | 95.23 | |
| GrftIntervention Layer=7-th layer2025.02 | 98.23 | 97.03 | |
| Grft-requeryIntervention Layer=7-th layer, Evaluation Protocol=Re-querying2025.02 | 97.3 | 96.99 | |
| LLMIntervention Layer=7-th layer2025.02 | 97.27 | 97.09 | |
| LLMBackbone=Llama-32025.02 | 96.89 | 97.52 | |
| System PromptIntervention Layer=7-th layer2025.02 | 96.85 | 95.72 | |
| FT-Llama-FullIntervention Layer=7-th layer, Fine-tuning Strategy=Full2025.02 | 96.29 | 93.08 | |
| ICLBackbone=Llama-32025.02 | 96.1 | 97.69 | |
| ReFT-GateBackbone=Llama-3, Alias=Grft2025.02 | 95.99 | 97.49 | |
| FT-Llama-LoraIntervention Layer=7-th layer, Fine-tuning Strategy=LoRA2025.02 | 95.52 | 89.79 | |
| ReFT-Gate-re-queryBackbone=Llama-3, Alias=Grft-requery2025.02 | 95.38 | 97.51 | |
| FT-Llama-FullBackbone=Llama-32025.02 | 95.26 | 94.39 | |
| FT-Llama-LoraBackbone=Llama-32025.02 | 94.32 | 96.03 | |
| Grft-W/O LossIntervention Layer=7-th layer, Loss=Disabled2025.02 | 92.36 | 90.17 | |
| ReFT-Gate-W/O lossBackbone=Llama-32025.02 | 92.09 | 91.05 | |
| ReFT(Training)Backbone=Llama-32025.02 | 91.18 | 89.07 | |
| System PromptBackbone=Llama-32025.02 | 89.33 | 98.08 | |
| Grft-W/O GateIntervention Layer=7-th layer, Gate=Disabled2025.02 | 89.03 | 88.36 | |
| Astute-RAGBackbone=Llama-32025.02 | 81.66 | 81.46 | |
| ICLIntervention Layer=7-th layer, Evaluation Protocol=In-Context Learning2025.02 | 80.81 | 92 | |
| Astute-RAGIntervention Layer=7-th layer2025.02 | 72.88 | 86.73 |