Confidence Estimation (Iterative Tagging) on PopQA
9.6Brier ScoreLOVEC-DPO
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LOVEC-DPOTagging Format=Iterative, Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 9.6 | 1.7 | 63.1 | |
| LOVEC-GRPOTagging Format=Iterative, Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 11.3 | 6.2 | 62.8 | |
| LOVEC-DPOBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 11.6 | 13.2 | 65.3 | |
| LOVEC-GRPOBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 13.1 | 18.1 | 64.2 | |
| Self-ConsBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 13.5 | 16.5 | 48.5 | |
| luqBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 13.6 | 15.1 | 62.6 | |
| p(true)-ftBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 14.1 | 12 | 42.2 | |
| LUQBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 16.7 | 23.2 | 62.5 | |
| Self-ConsBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 17.3 | 21.6 | 56.8 | |
| LOVEC-SFTTagging Format=Iterative, Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 18 | 25.9 | 52.7 | |
| Verb-ConfBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 18.8 | 22.1 | 18.3 | |
| Verb-ConfBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 19.5 | 20 | 36.2 | |
| p(true)Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 19.9 | 24.3 | 23.1 | |
| p(true)Base Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 21.5 | 24.5 | 26 | |
| VanillaBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 21.7 | 23.7 | 4.9 | |
| VanillaBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 24 | 27.7 | 23.7 | |
| LOVEC-SFTBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 25.8 | 32.6 | 37.1 |