Multiple-choice task on ECARE KRE
82.38AccuracyIRCAN-CAD
Evaluation results
| Method | Links | ||
|---|---|---|---|
| IRCAN-CADBackbone=Gemma-2B-it2024.06 | 82.38 | 17.62 | |
| IRCAN-CADBackbone=LLaMA-2-7B-Chat2024.06 | 80.96 | 19.04 | |
| IRCAN-CADBackbone=LLaMA-2-13B-Chat2024.06 | 78.42 | 21.58 | |
| CADBackbone=LLaMA-2-13B-Chat2024.06 | 78.13 | 21.87 | |
| CADBackbone=Gemma-2B-it2024.06 | 77.76 | 22.24 | |
| IRCANBackbone=Gemma-2B-it2024.06 | 77.38 | 22.62 | |
| Based_on_FormattedBackbone=Gemma-2B-it2024.06 | 76.72 | 23.28 | |
| Utilizing_FormattedBackbone=Gemma-2B-it2024.06 | 76.44 | 23.56 | |
| IRCANBackbone=LLaMA-2-7B-Chat2024.06 | 76.06 | 23.94 | |
| Based_onBackbone=Gemma-2B-it2024.06 | 75.59 | 24.41 | |
| OriginalBackbone=Gemma-2B-it2024.06 | 75.49 | 24.51 | |
| ITI (Mass Mean Shift)Backbone=Gemma-2B-it2024.06 | 73.8 | 26.2 | |
| OpinBackbone=LLaMA-3-8B-Instruct2024.06 | 73.8 | 26.2 | |
| CADBackbone=LLaMA-2-7B-Chat2024.06 | 73.7 | 23.3 | |
| ITI (Probe Weight Direction)Backbone=LLaMA-2-13B-Chat2024.06 | 73.42 | 26.58 | |
| ITI (Probe Weight Direction)Backbone=Gemma-2B-it2024.06 | 73.04 | 26.96 | |
| ITI (Mass Mean Shift)Backbone=LLaMA-2-13B-Chat2024.06 | 71.44 | 28.56 | |
| IRCANBackbone=LLaMA-2-13B-Chat2024.06 | 69.84 | 30.16 | |
| CADBackbone=LLaMA-3-8B-Instruct2024.06 | 69.56 | 30.44 | |
| IRCAN-CADBackbone=LLaMA-3-8B-Instruct2024.06 | 69.37 | 30.63 | |
| Based_on_FormattedBackbone=LLaMA-3-8B-Instruct2024.06 | 69.09 | 30.91 | |
| Utilizing_FormattedBackbone=LLaMA-3-8B-Instruct2024.06 | 68.99 | 31.01 | |
| OriginalBackbone=LLaMA-2-13B-Chat2024.06 | 68.9 | 31.1 | |
| Based_on_FormattedBackbone=LLaMA-2-13B-Chat2024.06 | 68.61 | 31.39 | |
| Based_onBackbone=LLaMA-2-13B-Chat2024.06 | 67.86 | 32.14 | |
| Utilizing_FormattedBackbone=LLaMA-2-13B-Chat2024.06 | 66.16 | 33.84 | |
| OpinBackbone=Gemma-2B-it2024.06 | 63.52 | 36.48 | |
| Based_on_FormattedBackbone=LLaMA-2-7B-Chat2024.06 | 63.15 | 36.85 | |
| Based_onBackbone=LLaMA-2-7B-Chat2024.06 | 61.55 | 38.45 | |
| Utilizing_FormattedBackbone=LLaMA-2-7B-Chat2024.06 | 60.98 | 39.02 | |
| ITI (Probe Weight Direction)Backbone=LLaMA-3-8B-Instruct2024.06 | 60.51 | 39.49 | |
| Based_onBackbone=LLaMA-3-8B-Instruct2024.06 | 59.1 | 40.9 | |
| IRCANBackbone=LLaMA-3-8B-Instruct2024.06 | 57.87 | 42.13 | |
| OriginalBackbone=LLaMA-3-8B-Instruct2024.06 | 57.4 | 42.6 | |
| OpinBackbone=LLaMA-2-13B-Chat2024.06 | 57.12 | 42.88 | |
| OriginalBackbone=LLaMA-2-7B-Chat2024.06 | 55.04 | 44.96 | |
| OpinBackbone=LLaMA-2-7B-Chat2024.06 | 55.04 | 44.96 | |
| ITI (Probe Weight Direction)Backbone=LLaMA-2-7B-Chat2024.06 | 49.58 | 50.42 | |
| ITI (Mass Mean Shift)Backbone=LLaMA-3-8B-Instruct2024.06 | 49.58 | 50.42 | |
| ITI (Mass Mean Shift)Backbone=LLaMA-2-7B-Chat2024.06 | 47.6 | 52.4 |