Output Consistency on Subject and Property Perturbations
91Subject AgreementGPT-4
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPT-4Year=2023, Prompt Prefix=Answer with the name only2025.01 | 91 | 94 | |
| ChatGPTYear=2022, Prompt Prefix=Answer with the name only2025.01 | 89 | 98 | |
| MistralI.Year=2023, Prompt Prefix=Answer with the name only, Tuning=Instruct2025.01 | 76 | 87 | |
| Llama-3I.Year=2024, Prompt Prefix=Answer with the name only, Tuning=Instruct2025.01 | 76 | 84 | |
| Llama-2C.Year=2023, Prompt Prefix=Answer with the name only, Tuning=Chat2025.01 | 72 | 82 | |
| MixtralI.Year=2023, Prompt Prefix=Answer with the name only, Tuning=Instruct2025.01 | 67 | 88 | |
| MistralYear=2023, Prompt Prefix=None2025.01 | 57 | 34 | |
| FalconI.Year=2023, Prompt Prefix=Answer with the name only, Tuning=Instruct2025.01 | 56 | 66 | |
| VicunaYear=2023, Prompt Prefix=Answer with the name only2025.01 | 56 | 69 | |
| MixtralYear=2023, Prompt Prefix=None2025.01 | 50 | 29 | |
| Flan-T5Year=2022, Prompt Prefix=None2025.01 | 48 | 49 | |
| OpenELM 3BYear=2024, Prompt Prefix=None2025.01 | 47 | 27 | |
| FalconYear=2023, Prompt Prefix=None2025.01 | 46 | 31 | |
| BloomYear=2022, Prompt Prefix=None2025.01 | 41 | 25 | |
| Llama-2Year=2023, Prompt Prefix=None2025.01 | 39 | 24 | |
| OLMo 7BYear=2024, Prompt Prefix=None2025.01 | 39 | 23 | |
| Llama-3Year=2024, Prompt Prefix=None2025.01 | 32 | 25 | |
| GPT-JYear=2021, Prompt Prefix=None2025.01 | 31 | 25 | |
| GPT-2Year=2019, Prompt Prefix=None2025.01 | 30 | 11 | |
| OpenELM 1.1BYear=2024, Prompt Prefix=None2025.01 | 29 | 22 | |
| OLMo 1BYear=2024, Prompt Prefix=None2025.01 | 28 | 20 | |
| GPT-3Year=2020, Prompt Prefix=None2025.01 | 10 | 9 | |
| T5Year=2020, Prompt Prefix=None2025.01 | 10 | 19 | |
| OpenELM 270MYear=2024, Prompt Prefix=None2025.01 | 6 | 4 |