Selective Question Answering on TriviaQA (test)
0Fail RateCIC
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 6.4 | 16.3 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.4 | 16.3 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.3 | 16.3 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.3 | 16.3 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.6 | 16.3 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 13.6 | 16.3 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.3 | 16.3 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.3 | 16.3 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 7.7 | 12.5 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.5 | 12.5 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.5 | 12.5 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.5 | 12.5 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 9.1 | 12.5 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.5 | 12.5 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.5 | 12.5 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.5 | 12.5 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 7.8 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.8 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.8 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.8 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-CP, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 3.9 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-CP, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 9.2 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-CP, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.8 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-CP, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.8 | 11.8 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-CP, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.8 | 11.8 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 6.5 | 16.9 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 12.6 | 16.9 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.9 | 16.9 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.9 | 16.9 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.6 | 16.9 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 13.8 | 16.9 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.9 | 16.9 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 16.9 | 16.9 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 7.8 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.5 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.5 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.5 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 4.2 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 9 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.5 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.5 | 11.5 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.5 | 11.5 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 7.7 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.6 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.6 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.6 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 3.7 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 9 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.6 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.6 | 11.6 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 11.6 | 11.6 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 7.9 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.8 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.8 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.8 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 4 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.1, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.8 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.15, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.8 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.2, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.8 | 8.8 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.25, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 0 | 8.8 | 8.8 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 4 | 3.7 | 12.5 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-CP, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 34 | 3.8 | 16.3 | |
| CICLLM=LLaMA-3.1-8B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 52 | 2.5 | 11.5 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-CP, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 99 | 5.7 | 16.9 | |
| CICLLM=Qwen2.5-3B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 100 | — | 16.3 | |
| CICLLM=Qwen2.5-7B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 100 | — | 12.5 | |
| CICLLM=OpenChat-3.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 100 | — | 11.8 | |
| CICLLM=Vicuna-7b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 100 | — | 16.9 | |
| CICLLM=Vicuna-13b-v1.5, UCB Type=UCB-HFD, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 100 | — | 11.6 | |
| CICLLM=Qwen2.5-14B-Instruct, UCB Type=UCB-HFD, Risk Level (alpha)=0.05, Semantic entropy samples=10, Sentence similarity threshold=0.62026.07 | 100 | — | 8.8 |