Confidence Calibration in Retrieval-Augmented Generation on Bamboogle k=5 OOD (test)
0.065ECENAACL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| NAACLBackbone=Qwen2.5-7B-Instruct, Number of retrieved passages (k)=52026.01 | 0.065 | 0.845 | |
| CoTBackbone=Qwen2.5-7B-Instruct, Number of retrieved passages (k)=52026.01 | 0.121 | 0.81 | |
| VanillaBackbone=Qwen2.5-7B-Instruct, Number of retrieved passages (k)=52026.01 | 0.126 | 0.835 | |
| NAACLBackbone=Llama-3.1-8B-Instruct, Number of retrieved passages (k)=52026.01 | 0.127 | 0.823 | |
| EnsembleBackbone=Qwen2.5-7B-Instruct, Number of retrieved passages (k)=52026.01 | 0.127 | 0.831 | |
| NAACLBackbone=DeepSeek-R1-Distill-Qwen-7B, Number of retrieved passages (k)=52026.01 | 0.127 | 0.765 | |
| Label-only SFTBackbone=Qwen2.5-7B-Instruct, Number of retrieved passages (k)=52026.01 | 0.129 | 0.658 | |
| Noise-awareBackbone=Qwen2.5-7B-Instruct, Number of retrieved passages (k)=52026.01 | 0.135 | 0.753 | |
| NAACLBackbone=DeepSeek-R1-Distill-Llama-8B, Number of retrieved passages (k)=52026.01 | 0.137 | 0.697 | |
| Noise-awareBackbone=Llama-3.1-8B-Instruct, Number of retrieved passages (k)=52026.01 | 0.14 | 0.806 | |
| EnsembleBackbone=DeepSeek-R1-Distill-Llama-8B, Number of retrieved passages (k)=52026.01 | 0.143 | 0.711 | |
| Noise-awareBackbone=DeepSeek-R1-Distill-Qwen-7B, Number of retrieved passages (k)=52026.01 | 0.143 | 0.625 | |
| Label-only SFTBackbone=Llama-3.1-8B-Instruct, Number of retrieved passages (k)=52026.01 | 0.151 | 0.721 | |
| Noise-awareBackbone=DeepSeek-R1-Distill-Llama-8B, Number of retrieved passages (k)=52026.01 | 0.153 | 0.658 | |
| VanillaBackbone=DeepSeek-R1-Distill-Llama-8B, Number of retrieved passages (k)=52026.01 | 0.154 | 0.671 | |
| EnsembleBackbone=Llama-3.1-8B-Instruct, Number of retrieved passages (k)=52026.01 | 0.173 | 0.68 | |
| EnsembleBackbone=DeepSeek-R1-Distill-Qwen-7B, Number of retrieved passages (k)=52026.01 | 0.198 | 0.55 | |
| CoTBackbone=Llama-3.1-8B-Instruct, Number of retrieved passages (k)=52026.01 | 0.199 | 0.579 | |
| CoTBackbone=DeepSeek-R1-Distill-Llama-8B, Number of retrieved passages (k)=52026.01 | 0.203 | 0.633 | |
| VanillaBackbone=Llama-3.1-8B-Instruct, Number of retrieved passages (k)=52026.01 | 0.212 | 0.633 | |
| Label-only SFTBackbone=DeepSeek-R1-Distill-Qwen-7B, Number of retrieved passages (k)=52026.01 | 0.216 | 0.65 | |
| CoTBackbone=DeepSeek-R1-Distill-Qwen-7B, Number of retrieved passages (k)=52026.01 | 0.218 | 0.638 | |
| VanillaBackbone=DeepSeek-R1-Distill-Qwen-7B, Number of retrieved passages (k)=52026.01 | 0.22 | 0.626 | |
| Label-only SFTBackbone=DeepSeek-R1-Distill-Llama-8B, Number of retrieved passages (k)=52026.01 | 0.251 | 0.684 |