Contextual Robustness Question Answering on ConflictQA (Known queries)
82.49Accuracy (Contradictory Short)Grft-requery
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Grft-requeryIntervention Layer=7-th layer, Evaluation Protocol=Re-querying2025.02 | 82.49 | 88.15 | 97.68 | 98.46 | 99.38 | |
| ReFT-Gate-re-queryBackbone=Llama-3, Alias=Grft-requery2025.02 | 69.47 | 78.18 | 82.02 | 96.03 | 99.71 | |
| GrftIntervention Layer=7-th layer2025.02 | 60.88 | 61.19 | 73.22 | 68.86 | 99.07 | |
| Grft-W/O GateIntervention Layer=7-th layer, Gate=Disabled2025.02 | 60.42 | 45.39 | 72.99 | 67.7 | 94.13 | |
| Astute-RAGIntervention Layer=7-th layer2025.02 | 59.6 | 46.7 | 72.56 | 69.8 | 93.6 | |
| ReFT-GateBackbone=Llama-3, Alias=Grft2025.02 | 54.11 | 52.25 | 70.86 | 66.36 | 98.04 | |
| Astute-RAGBackbone=Llama-32025.02 | 53.44 | 46.55 | 69.8 | 73.66 | 94.5 | |
| COTIntervention Layer=7-th layer, Evaluation Protocol=Chain-of-Thought2025.02 | 41.83 | 36.02 | 42.68 | 36.25 | 99.04 | |
| Grft-W/O LossIntervention Layer=7-th layer, Loss=Disabled2025.02 | 41.05 | 36.48 | 72.3 | 68.09 | 98.14 | |
| ReFT(Training)Backbone=Llama-32025.02 | 40.08 | 43.05 | 62.03 | 69.01 | 93.02 | |
| ReFT-Gate-W/O lossBackbone=Llama-32025.02 | 39.33 | 44.07 | 63.06 | 68.85 | 95.9 | |
| System PromptBackbone=Llama-32025.02 | 36.36 | 35.02 | 59.19 | 47.08 | 98.37 | |
| System PromptIntervention Layer=7-th layer2025.02 | 35.48 | 25.56 | 53.68 | 44.85 | 91.87 | |
| LLMIntervention Layer=7-th layer2025.02 | 34.55 | 25.33 | 53.14 | 44.62 | 99.26 | |
| FT-Llama-FullIntervention Layer=7-th layer, Fine-tuning Strategy=Full2025.02 | 32.68 | 26.94 | 54.08 | 47.28 | 95.29 | |
| FT-Llama-LoraBackbone=Llama-3, Fine-tuning=LoRA2025.02 | 32.08 | 30.18 | 26.09 | 25.89 | 93.28 | |
| FT-Llama-FullBackbone=Llama-3, Fine-tuning=Full2025.02 | 31.98 | 29.02 | 27.19 | 24.96 | 95.03 | |
| FT-Llama-LoraIntervention Layer=7-th layer, Fine-tuning Strategy=LoRA2025.02 | 31.68 | 27.42 | 52.21 | 47.25 | 97.68 | |
| COTBackbone=Llama-32025.02 | 30.81 | 26.08 | 19 | 19 | 99.57 | |
| ICLBackbone=Llama-32025.02 | 29.19 | 27.89 | 25.31 | 24.78 | 97.66 | |
| LLMBackbone=Llama-32025.02 | 26.47 | 26.99 | 51.67 | 39.62 | 99.67 | |
| ICLIntervention Layer=7-th layer, Evaluation Protocol=In-Context Learning2025.02 | 21.07 | 23.01 | 22.77 | 24.94 | 80.79 |