Input-bound consistency on Subject and Property Perturbations dataset
38Subject Agreement (%)ENtity-Aware Finetuning
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ENtity-Aware FinetuningModel=GPT-2, Representation=ID Tags (Subject)2025.01 | 38 | 22 | |
| ENtity-Aware FinetuningModel=OLMo 1B, Representation=Normalized Entity Representation (Subject & Attribute)2025.01 | 33 | 9 | |
| ENtity-Aware FinetuningModel=GPT-2, Representation=ID Tags (Subject) + Selected Named Entity Tags (Attribute)2025.01 | 31 | 28 | |
| ENtity-Aware FinetuningModel=GPT-2, Representation=Normalized Entity Representation (Subject & Attribute)2025.01 | 29 | 33 | |
| Pre-Trained ModelModel=GPT-22025.01 | 28 | 24 | |
| ENtity-Aware FinetuningModel=OLMo 1B, Representation=ID Tags (Subject) + Selected Named Entity Tags (Attribute)2025.01 | 22 | 0 | |
| ENtity-Aware FinetuningModel=GPT-2, Representation=Selected Named Entity Tags (Subject & Attribute)2025.01 | 18 | 26 | |
| Pre-Trained ModelModel=OLMo 1B2025.01 | 13 | 44 | |
| Vanilla FinetuningModel=GPT-22025.01 | 12 | 30 | |
| Vanilla FinetuningModel=OLMo 1B2025.01 | 11 | 33 | |
| ENtity-Aware FinetuningModel=OLMo 1B, Representation=ID Tags (Subject)2025.01 | 8 | 6 | |
| ENtity-Aware FinetuningModel=OLMo 1B, Representation=Named Entity Tags (Subject & Attribute)2025.01 | 6 | 35 | |
| ENtity-Aware FinetuningModel=OLMo 1B, Representation=Selected Named Entity Tags (Subject & Attribute)2025.01 | 6 | 43 | |
| ENtity-Aware FinetuningModel=GPT-2, Representation=Named Entity Tags (Subject & Attribute)2025.01 | 0 | 50 |