List your company

Input-bound consistency on Subject and Property Perturbations dataset

38Subject Agreement (%)ENtity-Aware Finetuning
-1.528.741929.26Jan 22, 2025

Evaluation results

MethodLinks
ENtity-Aware FinetuningModel=GPT-2, Representation=ID Tags (Subject)2025.013822
ENtity-Aware FinetuningModel=OLMo 1B, Representation=Normalized Entity Representation (Subject & Attribute)2025.01339
ENtity-Aware FinetuningModel=GPT-2, Representation=ID Tags (Subject) + Selected Named Entity Tags (Attribute)2025.013128
ENtity-Aware FinetuningModel=GPT-2, Representation=Normalized Entity Representation (Subject & Attribute)2025.012933
Pre-Trained ModelModel=GPT-22025.012824
ENtity-Aware FinetuningModel=OLMo 1B, Representation=ID Tags (Subject) + Selected Named Entity Tags (Attribute)2025.01220
ENtity-Aware FinetuningModel=GPT-2, Representation=Selected Named Entity Tags (Subject & Attribute)2025.011826
Pre-Trained ModelModel=OLMo 1B2025.011344
Vanilla FinetuningModel=GPT-22025.011230
Vanilla FinetuningModel=OLMo 1B2025.011133
ENtity-Aware FinetuningModel=OLMo 1B, Representation=ID Tags (Subject)2025.0186
ENtity-Aware FinetuningModel=OLMo 1B, Representation=Named Entity Tags (Subject & Attribute)2025.01635
ENtity-Aware FinetuningModel=OLMo 1B, Representation=Selected Named Entity Tags (Subject & Attribute)2025.01643
ENtity-Aware FinetuningModel=GPT-2, Representation=Named Entity Tags (Subject & Attribute)2025.01050