Confidence Estimation (Iterative Tagging) on Bios
7.5Brier Score (BS)LOVEC-DPO
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LOVEC-DPOBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 7.5 | 7.3 | 75.2 | |
| LOVEC-GRPOTagging Format=Iterative, Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 8.5 | 4.2 | 64.7 | |
| LOVEC-DPOTagging Format=Iterative, Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 9 | 7.3 | 65.6 | |
| LOVEC-GRPOBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 10.7 | 11.1 | 72.4 | |
| luqBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 12.2 | 15.5 | 69.2 | |
| Self-ConsBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 13 | 17 | 53 | |
| p(true)-ftBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 15.2 | 21.7 | 39.8 | |
| LOVEC-SFTTagging Format=Iterative, Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 16.6 | 25.8 | 56 | |
| Verb-ConfBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 18 | 19 | 39.5 | |
| LOVEC-SFTBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 18.8 | 25.1 | 54.9 | |
| p(true)Backbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 19.7 | 28.6 | 17.3 | |
| LUQBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 20 | 29.5 | 63.8 | |
| Self-ConsBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 20.3 | 26.5 | 58.8 | |
| VanillaBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 20.9 | 24.1 | 1.2 | |
| p(true)Base Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 21 | 25 | 31 | |
| Verb-ConfBackbone=Llama3-8B-Instruct, Decoding Strategy=Greedy2025.05 | 21.2 | 25.3 | 10.8 | |
| VanillaBase Model=Gemma-2-9B-It, Tagging Strategy=Sentence-level iterative tagging2025.05 | 24.5 | 28.8 | 35.5 |