# Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction

Source: [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](<https://www.sota2.com/research/paper/68fa6ad11e209131cea2a835>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:10:35.496Z

Authors: Shauli Ravfogel, Grusha Prasad, Tal Linzen, Yoav Goldberg

Published: 2021-05-14T17:11:55.000Z

## Abstract

When language models process syntactically complex sentences, do they use their representations of syntax in a manner that is consistent with the grammar of the language? We propose AlterRep, an intervention-based method to address this question. For any linguistic feature of a given sentence, AlterRep generates counterfactual representations by altering how the feature is encoded, while leaving intact all other aspects of the original representation. By measuring the change in a model's word prediction behavior when these counterfactual representations are substituted for the original ones, we can draw conclusions about the causal effect of the linguistic feature in question on the model's behavior. We apply this method to study how BERT models of different sizes process relative clauses (RCs). We find that BERT variants use RC boundary information during word prediction in a manner that is consistent with the rules of English grammar; this RC boundary information generalizes to a considerable extent across different RC types, suggesting that BERT represents RCs as an abstract linguistic category.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Lexical Gender Dataset evaluation on LGD](<https://www.sota2.com/research/sota/lexical-gender-dataset-evaluation-on-lgd.md>): Lexical Gender Dataset evaluation; LGD; Completeness: 99.51.

- [LGD on LGD](<https://www.sota2.com/research/sota/lgd-on-lgd.md>): LGD; LGD; Completeness: 99.51.

- [Gender agreement on CausalGym agr\_gender](<https://www.sota2.com/research/sota/gender-agreement-on-causalgym-agr_gender.md>): Gender agreement; CausalGym agr\_gender; Completeness: 100.

- [NPI licensing in subject-relative clauses on CausalGym npi\_any\_subj-relc](<https://www.sota2.com/research/sota/npi-licensing-in-subject-relative-clauses-on-causalgym-npi_any_subj-relc.md>): NPI licensing in subject-relative clauses; CausalGym npi\_any\_subj-relc; Completeness: 100.

- [agr\_refl\_num\_pp on CausalGym](<https://www.sota2.com/research/sota/agr_refl_num_pp-on-causalgym.md>): agr\_refl\_num\_pp; CausalGym; Completeness: 99.

- [Filler Gap Subject Dependency on CausalGym](<https://www.sota2.com/research/sota/filler-gap-subject-dependency-on-causalgym.md>): filler\_gap\_subj; CausalGym; Completeness: 100.

- [gss\_subord\_subj-relc on CausalGym](<https://www.sota2.com/research/sota/gss_subord_subj-relc-on-causalgym.md>): gss\_subord\_subj-relc; CausalGym; Completeness: 97.

- [Agreement, Reflexive, Number, Subject-Relative Clause on CausalGym agr\_refl\_num\_subj-relc](<https://www.sota2.com/research/sota/agreement-reflexive-number-subject-relative-clause-on-causalgym-agr_refl_num_subj-relc.md>): agr\_refl\_num\_subj-relc; CausalGym; Completeness: 98.98.

- [filler\_gap\_pp on CausalGym](<https://www.sota2.com/research/sota/filler_gap_pp-on-causalgym.md>): filler\_gap\_pp; CausalGym; Completeness: 80.95.

- [gss\_subord\_pp on CausalGym](<https://www.sota2.com/research/sota/gss_subord_pp-on-causalgym.md>): gss\_subord\_pp; CausalGym; Completeness: 97.

- [agr\_sv\_num\_pp on CausalGym](<https://www.sota2.com/research/sota/agr_sv_num_pp-on-causalgym.md>): agr\_sv\_num\_pp; CausalGym; Completeness: 95.

- [Cleft sentence structure analysis on CausalGym cleft](<https://www.sota2.com/research/sota/cleft-sentence-structure-analysis-on-causalgym-cleft.md>): Cleft sentence structure analysis; CausalGym cleft; Completeness: 86.03.

- [Filler-gap dependency hierarchy on CausalGym filler\_gap\_hierarchy](<https://www.sota2.com/research/sota/filler-gap-dependency-hierarchy-on-causalgym-filler_gap_hierarchy.md>): Filler-gap dependency hierarchy; CausalGym filler\_gap\_hierarchy; Completeness: 99.

- [Subject-verb number agreement in object-relative clauses on CausalGym agr\_sv\_num\_obj-relc](<https://www.sota2.com/research/sota/subject-verb-number-agreement-in-object-relative-clauses-on-causalgym-agr_sv_num_obj-relc.md>): Subject-verb number agreement in object-relative clauses; CausalGym agr\_sv\_num\_obj-relc; Completeness: 98.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2105.06965>)

[PDF](<https://arxiv.org/pdf/2105.06965.pdf>)