# Pronoun Disambiguation on Winograd Schema Challenge Source: [Pronoun Disambiguation on Winograd Schema Challenge]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:27:54.372Z Evaluation of model performance on resolving pronoun ambiguities using the standard Winograd Schema Challenge dataset, typically measured by accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Pronoun Disambiguation | | Dataset (including any stated split) | Winograd Schema Challenge | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 27 of 27 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | RoBERTa-WG | 90.1 | Not recorded | 68fa6a031e209131cea256d1 | | CorefRoBERTa LARGE | 83.2 | Not recorded | 68fa6a031e209131cea256d1 | | RoBERTa-DPR | 83.1 | Not recorded | 68fa6a031e209131cea256d1 | | MNPP | 79.5 | Not recorded | 68fa6a031e209131cea256d1 | | AMS-CSQA-DPR | 75.5 | Not recorded | 68fa6a031e209131cea256d1 | | UDSSM-MASKEDWIKI-DPR | 75.1 | Not recorded | 68fa6a031e209131cea256d1 | | BERT-MASKEDWIKI-DPR | 72.5 | Not recorded | 68fa6a031e209131cea256d1 | | BERT-WIKICREM-DPR | 71.8 | Not recorded | 68fa6a031e209131cea256d1 | | CorefBERT LARGE | 71.4 | Not recorded | 68fa6a031e209131cea256d1 | | BERT\_NSP-DPR | 71.1 | Not recorded | 68fa6a031e209131cea256d1 | | WSC Know. Hunting | 71.1 | Not recorded | 68fa6a031e209131cea256d1 | | GPT-2 | 70.7 | Not recorded | 68fa6a031e209131cea256d1 | | CSS | 69.6 | Not recorded | 68fa6a031e209131cea256d1 | | Ensemble LMs | 63.7 | Not recorded | 68fa6a031e209131cea256d1 | | BERT-WIKICREAM | 63.4 | Not recorded | 68fa6a031e209131cea256d1 | | UDSSM | 62.4 | Not recorded | 68fa6a031e209131cea256d1 | | BERT-MASKEDWIKI | 61.9 | Not recorded | 68fa6a031e209131cea256d1 | | Maximum Attention Score (MAS) | 60.3 | Not recorded | 68fa65dc1e209131cea0ce64 | | Modified Skip-Gram | 60.3 | Not recorded | 68fa6a031e209131cea256d1 | | BERT Inner Attention | 60.3 | Not recorded | 68fa6a031e209131cea256d1 | | Know. Hunter | 57.1 | Not recorded | 68fa65dc1e209131cea0ce64 | | Bi-LSTM-DPR | 56 | Not recorded | 68fa6a031e209131cea256d1 | | Single LM | 54.5 | Not recorded | 68fa65dc1e209131cea0ce64 | | Transformer | 54.1 | Not recorded | 68fa65dc1e209131cea0ce64 | | USSM + Supervised DeepNet + KB | 52.8 | Not recorded | 68fa65dc1e209131cea0ce64 | | USSM + KB | 52 | Not recorded | 68fa65dc1e209131cea0ce64 | | Random guess | 50 | Not recorded | 68fa65dc1e209131cea0ce64 | ### RoBERTa-WG Accuracy: 90.1. Result ID: 698dfe2cffc3901de09a01d3. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"WinoGrande","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### CorefRoBERTa LARGE Accuracy: 83.2. Result ID: 698dfe2cffc3901de09a01d2. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### RoBERTa-DPR Accuracy: 83.1. Result ID: 698dfe2cffc3901de09a01d1. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"DPR","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### MNPP Accuracy: 79.5. Result ID: 698dfe2cffc3901de09a01dd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### AMS-CSQA-DPR Accuracy: 75.5. Result ID: 698dfe2cffc3901de09a01d0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"CSQA, DPR","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### UDSSM-MASKEDWIKI-DPR Accuracy: 75.1. Result ID: 698dfe2cffc3901de09a01cf. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"MASKEDWIKI, DPR","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### BERT-MASKEDWIKI-DPR Accuracy: 72.5. Result ID: 698dfe2cffc3901de09a01ce. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"MASKEDWIKI, DPR","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### BERT-WIKICREM-DPR Accuracy: 71.8. Result ID: 698dfe2cffc3901de09a01cd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"WIKICREM, DPR","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### CorefBERT LARGE Accuracy: 71.4. Result ID: 698dfe2cffc3901de09a01cc. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### BERT\_NSP-DPR Accuracy: 71.1. Result ID: 698dfe2cffc3901de09a01cb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"DPR","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### WSC Know. Hunting Accuracy: 71.1. Result ID: 698dfe2cffc3901de09a01dc. [Reporting paper]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### GPT-2 Accuracy: 70.7. Result ID: 698dfe2cffc3901de09a01db. [Reporting paper]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### CSS Accuracy: 69.6. Result ID: 698dfe2cffc3901de09a01da. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### Ensemble LMs Accuracy: 63.7. Result ID: 698dfe2cffc3901de09a01d9. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### BERT-WIKICREAM Accuracy: 63.4. Result ID: 698dfe2cffc3901de09a01d8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### UDSSM Accuracy: 62.4. Result ID: 698dfe2cffc3901de09a01d7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### BERT-MASKEDWIKI Accuracy: 61.9. Result ID: 698dfe2cffc3901de09a01d6. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### Maximum Attention Score (MAS) Accuracy: 60.3. Result ID: 698cce008b517eb4331cdf09. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"model":"BERT"}. This record does not assert independent reproduction. ### Modified Skip-Gram Accuracy: 60.3. Result ID: 698dfe2cffc3901de09a01d4. [Reporting paper]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### BERT Inner Attention Accuracy: 60.3. Result ID: 698dfe2cffc3901de09a01d5. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Supervision Type":"Unsupervised","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### Know. Hunter Accuracy: 57.1. Result ID: 698cce008b517eb4331cdf08. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Bi-LSTM-DPR Accuracy: 56. Result ID: 698dfe2cffc3901de09a01ca. [Reporting paper]() Recorded evaluation configuration: {"Supervision Type":"Supervised","Finetuning Dataset":"DPR","Zero-shot Transfer":"True"}. This record does not assert independent reproduction. ### Single LM Accuracy: 54.5. Result ID: 698cce008b517eb4331cdf06. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Transformer Accuracy: 54.1. Result ID: 698cce008b517eb4331cdf07. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### USSM + Supervised DeepNet + KB Accuracy: 52.8. Result ID: 698cce008b517eb4331cdf05. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### USSM + KB Accuracy: 52. Result ID: 698cce008b517eb4331cdf04. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Random guess Accuracy: 50. Result ID: 698cce008b517eb4331cdf03. [Reporting paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.