Grammatical Error Correction on CoNLL 2014 (test)
72.2F0.5 ScoreAGGR-RANK*
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AGGR-RANK*Ensemble Components=GPT-4-rank-a(clust 3), GRECO-rank-w(best 7), Optimized for test dataset=true2024.04 | 72.2 | 81.9 | 49 | — | — | — | — | — | — | — | — | — | — | — | |
| AGGR-RANKEnsemble Components=GPT-4-rank-a(clust 3), majority-voting(best 7)2024.04 | 71.8 | 84 | 45.4 | — | — | — | — | — | — | — | — | — | — | — | |
| GRECO_votingBase Systems=T5-XL, C4-200M, and models [2]-[5] from Table 32023.10 | 71.12 | 79.6 | 49.86 | — | — | — | — | — | — | — | — | — | — | — | |
| GRECO voting2023.10 | 70.48 | 79.36 | 48.69 | — | — | — | — | — | — | — | — | — | — | — | |
| GRECO voting+ESC2023.10 | 70.48 | 79.36 | 48.69 | — | — | — | — | — | — | — | — | — | — | — | |
| GRECOBase Systems=T5-XL, C4-200M, and models [2]-[5] from Table 32023.10 | 69.86 | 76.68 | 51.54 | — | — | — | — | — | — | — | — | — | — | — | |
| BART Baseline + CDA w/ denoisingModel Size=400M, Training Data=model-based synthetic (C4 200M), Contextual Data Augmentation=with denoising2024.06 | 69.8 | 76.2 | 52.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Zhou et al.Backbone=BART, Lang=Mono2026.06 | 69.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ESC2023.10 | 69.51 | 81.48 | 43.78 | — | — | — | — | — | — | — | — | — | — | — | |
| GRECO2023.10 | 69.23 | 76.39 | 50.35 | — | — | — | — | — | — | — | — | — | — | — | |
| Rothe et al. (2021)Model Category=Single models2022.03 | 68.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Rothe et al. (2021)model=T5, number of parameters=11B2022.04 | 68.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5-XXLModel Size=11B, Training Data=rule-based synthetic (1B word)2024.06 | 68.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MultiTaskBARTModel Size=400M, Training Data=rule-based synthetic (1B word), cleaned Lang82024.06 | 68.9 | 75.4 | 51.2 | — | — | — | — | — | — | — | — | — | — | — | |
| BART BaselineModel Size=400M, Training Data=model-based synthetic (C4 200M)2024.06 | 68.6 | 73.8 | 53.5 | — | — | — | — | — | — | — | — | — | — | — | |
| T5-XLModel Index=12023.10 | 68.5 | 74.4 | 52.02 | — | — | — | — | — | — | — | — | — | — | — | |
| MEMT2023.10 | 68.37 | 76.44 | 48.06 | — | — | — | — | — | — | — | — | — | — | — | |
| Stahlberg and Kumar (2021)Model Category=Ensembles2022.03 | 68.3 | 75.6 | 49.3 | — | — | — | — | — | — | — | — | — | — | — | |
| EditScorer2023.10 | 68.25 | 74.32 | 51.44 | — | — | — | — | — | — | — | — | — | — | — | |
| C4-200MModel Index=72023.10 | 68.13 | 75.47 | 49.06 | — | — | — | — | — | — | — | — | — | — | — | |
| TemplateGECParameters=125M+770M, Evaluation Protocol=Primary2024.05 | 68.1 | 74.8 | 50 | — | — | — | — | — | — | — | — | — | — | — | |
| TemplateGECModel Size=770M, Training Data=cleaned Lang8, model-based synthetic (300M)2024.06 | 68.1 | 74.8 | 50 | — | — | — | — | — | — | — | — | — | — | — | |
| DeCoGLMParameters=335M, Evaluation Protocol=Primary2024.05 | 68 | 75.1 | 49.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Kiyono et al. (2019)Synthetic data=true, Model composition=Ensemble2023.07 | 67.9 | 67.3 | 44 | — | — | — | — | — | — | — | — | — | — | — | |
| Chat-LLaMa-2-13Bfine-tuned=true2024.04 | 67.9 | 77.3 | 45.6 | — | — | — | — | — | — | — | — | — | — | — | |
| DeGLM-CoGLMParameters=335M+335M, Evaluation Protocol=Primary2024.05 | 67.8 | 75.1 | 49 | — | — | — | — | — | — | — | — | — | — | — | |
| T5-XLModel Size=3B, Training Data=rule-based synthetic (1B word)2024.06 | 67.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SynGECExtra Data Size=2.4M, Transformer Config=12+12,1024,4096, PLM=true, syntax=true2022.10 | 67.6 | 74.7 | 49 | — | — | — | — | — | — | — | — | — | — | — | |
| SynGECParameters=110M+400M, Evaluation Protocol=Primary2024.05 | 67.6 | 74.7 | 49 | — | — | — | — | — | — | — | — | — | — | — | |
| SynGECModel Size=400M, Training Data=rule-based synthetic (1B word), cleaned Lang82024.06 | 67.6 | 74.7 | 49 | — | — | — | — | — | — | — | — | — | — | — | |
| T5-11Bfine-tuned=true2024.04 | 67.5 | 70.9 | 56.5 | — | — | — | — | — | — | — | — | — | — | — | |
| UL2-20Bfine-tuned=true2024.04 | 67.5 | 73.8 | 50.4 | — | — | — | — | — | — | — | — | — | — | — | |
| MixEditModel Size=400M, Training Data=rule-based synthetic (1B word)2024.06 | 67.3 | 75.6 | 46.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Chat-LLaMa-2-7Bfine-tuned=true2024.04 | 67.2 | 75.5 | 46.8 | — | — | — | — | — | — | — | — | — | — | — | |
| BART Baseline + CDA w/o denoisingModel Size=400M, Training Data=model-based synthetic (C4 200M), Contextual Data Augmentation=without denoising2024.06 | 67.1 | 76.7 | 44.8 | — | — | — | — | — | — | — | — | — | — | — | |
| GOParExtra Data Size=2.4M, Transformer Config=12+12,1024,4096, parser=Off-the-shelf Parser, PLM=true, syntax=true2022.10 | 67 | 74.1 | 48.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Lichtarge et al. (2020)2022.04 | 66.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Our BaselineExtra Data Size=2.4M, Transformer Config=12+12,1024,4096, PLM=true, syntax=false2022.10 | 66.7 | 73.6 | 48.6 | — | — | — | — | — | — | — | — | — | — | — | |
| BARTParameters=400M, Evaluation Protocol=Primary2024.05 | 66.7 | 73.6 | 48.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Stahlberg and Kumar (2021)Model Category=Single models2022.03 | 66.6 | 72.8 | 49.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Stahlberg and Kumar (2021)Extra Data Size=540M, Transformer Config=12+12,1024,4096, PLM=false, syntax=false2022.10 | 66.6 | 72.8 | 49.5 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTORBackbone=BERT + ROBERTa + XLNet, Ensemble=true2020.05 | 66.5 | 78.2 | 41.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Omelianchuk et al. (2020)Model Category=Ensembles2022.03 | 66.5 | 78.2 | 41.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Omelianchuk et al. (2020)Synthetic data=true, Model composition=Ensemble2023.07 | 66.5 | 78.2 | 41.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Our approach (12+2 BART-Init)Synthetic Data=Yes, Multi-stage Fine-tuning=Yes2021.06 | 66.4 | 71 | 52.8 | — | — | — | — | — | — | 9.6 | — | — | — | — | |
| Sun et al. (2021)Extra Data Size=300M, Transformer Config=12+2,1024,4096, PLM=true, syntax=false2022.10 | 66.4 | 71 | 52.8 | — | — | — | — | — | — | — | — | — | — | — | |
| ShallowADModel Size=~240M, Training Data=model-based synthetic (300M)2024.06 | 66.4 | 71 | 52.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Rothe et al. (2021)Extra Data Size=2.4M, Transformer Config=12+12,1024,4096, PLM=true, syntax=false2022.10 | 66.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5Parameters=770M, Evaluation Protocol=Primary2024.05 | 66.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5-largeModel Size=770M, Training Data=rule-based synthetic (1B word)2024.06 | 66.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeGLM-CoGLMParameters=335M+10B, Evaluation Protocol=LoRA Fine-tuning2024.05 | 66.08 | 70.58 | 52.65 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTORBackbone=ROBERTa + XLNet, Ensemble=true2020.05 | 66 | 76.6 | 42.3 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTOR-2024fine-tuned=true2024.04 | 66 | 75 | 44.7 | — | — | — | — | — | — | — | — | — | — | — | |
| + BIFI (ours)Ensemble=false2021.09 | 65.8 | 78 | 40.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Rothe et al.Backbone=gT5 xxl, Lang=Mono2026.06 | 65.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CTC-Copy2024.04 | 65.5 | 72.6 | 47 | — | — | — | — | — | — | — | — | — | — | — | |
| EditScorer2024.04 | 65.5 | 78.5 | 39.4 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTOR XLNet2023.10 | 65.34 | 77.49 | 40.15 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA2Parameters=13B, Evaluation Protocol=LoRA Fine-tuning2024.05 | 65.33 | 68.43 | 55.3 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTORBackbone=XLNet, Ensemble=false2020.05 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| GECToR(XLNet)Synthetic Data=Yes, Multi-stage Fine-tuning=Yes2021.06 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTOR (Omelianchuk et al., 2020)Ensemble=false2021.09 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTOR (our base)Ensemble=false2021.09 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Omelianchuk et al. (2020)Model Category=Single models2022.03 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTa(L) + RoBERTa(L) + XLNet(L)Model Category=Ensembles2022.03 | 65.3 | 76.1 | 41.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Omelianchuk et al. (2020)Extra Data Size=9M, Transformer Config=12+0,768,3072, PLM=true, syntax=false2022.10 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Omelianchuk et al. (2020)Synthetic data=true, Model composition=Single2023.07 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| GECToRParameters=110M, Evaluation Protocol=Primary2024.05 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| GECToRModel Size=350M, Training Data=rule-based synthetic (1B word)2024.06 | 65.3 | 77.5 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Kaneko et al. (2020)Model Category=Ensembles2022.03 | 65.2 | 72.6 | 46.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Luhtaru et al.Backbone=NLLB, Lang=Multi2026.06 | 65.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Rothe et al. (2021)Synthetic data=false, Model composition=Single2023.07 | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5-Large2023.10 | 65.07 | 69.66 | 51.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Kiyono et al. (2019)Ensemble=true2020.05 | 65 | 72.4 | 46.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Kiyono et al. (2019)Ensemble=true2021.09 | 65 | 72.4 | 46.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Kiyono et al. (2019)Model Category=Ensembles2022.03 | 65 | 72.4 | 46.1 | — | — | — | — | — | — | — | — | — | — | — | |
| SOME2023.10 | 65 | 68.39 | 54.23 | — | — | — | — | — | — | — | — | — | — | — | |
| Riken&Tohoku2023.10 | 64.74 | 73.26 | 44.17 | — | — | — | — | — | — | — | — | — | — | — | |
| UEDIN-MS2023.10 | 64.52 | 75.15 | 41.21 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-22023.10 | 64.47 | 68.3 | 52.65 | — | — | — | — | — | — | — | — | — | — | — | |
| AMR-GECSynthetic data=false, Model composition=Single2023.07 | 64.4 | 70.2 | 48.3 | — | — | — | — | — | — | — | — | — | — | — | |
| AMR-GECSynthetic data=false, Model composition=Ensemble2023.07 | 64.4 | 70.3 | 48.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Grundkiewicz et al. (2019)Model Category=Ensembles2022.03 | 64.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Grundkiewicz et al. (2019)Ensemble=true, Training Data (W&I+L usage)=With W&I+L2019.10 | 64.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GECTORBackbone=ROBERTa, Ensemble=false2020.05 | 64 | 73.9 | 41.5 | — | — | — | — | — | — | — | — | — | — | — | |
| GECToR(RoBERTa)Synthetic Data=Yes, Multi-stage Fine-tuning=Yes2021.06 | 64 | 73.9 | 41.5 | — | — | — | — | — | — | 12.4 | — | — | — | — | |
| RoBERTa (L) 10KModel Category=Single models, Training Strategy=multi-stage training2022.03 | 64 | 74.4 | 41.05 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTOR ROBERTa2023.10 | 64 | 73.91 | 41.66 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4 + DeGLMEvaluation Protocol=Zero-shot with DeGLM detection2024.05 | 63.7 | 66.4 | 54.81 | — | — | — | — | — | — | — | — | — | — | — | |
| Our approach (9+3)Synthetic Data=Yes, Multi-stage Fine-tuning=No2021.06 | 63.5 | 73.3 | 41.3 | — | — | — | — | — | — | 10.3 | — | — | — | — | |
| SynGECExtra Data Size=2.4M, Transformer Config=6+6,512,2048, PLM=false, syntax=true2022.10 | 63.5 | 70 | 46.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Multi-EncoderParameters=110M+107M, Evaluation Protocol=Primary2024.05 | 63.5 | 71.3 | 44.3 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA2Parameters=7B, Evaluation Protocol=LoRA Fine-tuning2024.05 | 63.47 | 67.24 | 51.84 | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerEnsemble=false, Model variant=finetuned, Training Data (W&I+L usage)=With W&I+L2019.10 | 63.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Xu et al. (2019)Ensemble=true, Training Data (W&I+L usage)=With W&I+L2019.10 | 63.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Wan and Wan (2021)Extra Data Size=10M, Transformer Config=6+6,512,2048, PLM=false, syntax=true2022.10 | 63.2 | 74.4 | 39.5 | — | — | — | — | — | — | — | — | — | — | — | |
| GECTORBackbone=BERT, Ensemble=false2020.05 | 63 | 72.1 | 42 | — | — | — | — | — | — | — | — | — | — | — | |
| Katsumata and Komachi (2020)Synthetic data=false, Model composition=Ensemble2023.07 | 63 | 69.9 | 45.1 | — | — | — | — | — | — | — | — | — | — | — | |
| VERNet2023.10 | 62.85 | 74.08 | 39.12 | — | — | — | — | — | — | — | — | — | — | — | |
| Kaneko et al. (2020)Model Category=Single models2022.03 | 62.6 | 69.2 | 45.6 | — | — | — | — | — | — | — | — | — | — | — |