Grammatical Error Correction on BEA shared task 2019 (test)
80.84F0.5 ScoreGRECO voting+ESC
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GRECO voting+ESCBase systems=T5-XL + GECTOR-Large + models [2], [4], [5], [6] from Table 32023.10 | 80.84 | 86.66 | 63.72 | |
| GRECO voting+ESC2023.10 | 80.5 | 86.45 | 63.13 | |
| ESCBase systems=T5-XL + GECTOR-Large + models [2], [4], [5], [6] from Table 32023.10 | 80.1 | 86.64 | 61.54 | |
| EditScorer2023.10 | 80.05 | 88.05 | 58.71 | |
| Qorib et al.system combination=true2023.09 | 79.9 | 86.6 | 60.9 | |
| ESC2023.10 | 79.9 | 86.65 | 60.91 | |
| GRECO votingBase systems=T5-XL + GECTOR-Large + models [2], [4], [5], [6] from Table 32023.10 | 78.85 | 83.24 | 65.12 | |
| GRECO voting2023.10 | 78.58 | 82.86 | 65.1 | |
| 4xRPN ensembleensemble size=4, models=RPN(XLNBASE), RPN(XLNLARGE), RPN(MPRBASE), RPN(MPRLARGE)2023.09 | 77.6 | 86.62 | 54.8 | |
| MEMT2023.10 | 77.48 | 82.2 | 63 | |
| GRECOBase systems=T5-XL + GECTOR-Large + models [2], [4], [5], [6] from Table 32023.10 | 77.24 | 80.46 | 66.59 | |
| GRECO2023.10 | 76.83 | 80.03 | 66.22 | |
| GECToR Ensemble (DeBERTa + RoBERTa + XLNet)model_type=ensemble, backbones=DeBERTa-Large + RoBERTa-Large + XLNet-Large, ensemble_method=majority votes on span-level edits2022.03 | 76.05 | 84.44 | 54.42 | |
| Rothe et al.backbone=T5-XXL 11B2023.09 | 75.9 | — | — | |
| Rothe et al.model_type=single model2022.03 | 75.88 | — | — | |
| Stahlberg and Kumarmodel_type=ensemble2022.03 | 74.9 | 77.7 | 65.4 | |
| Stahlberg and Kumar2023.09 | 74.9 | 77.7 | 65.4 | |
| T5-XLConfiguration=Base system replacement for model [1]2023.10 | 74.72 | 76.85 | 67.26 | |
| RPN(MPRBASE)backbone=Muppet Roberta Base2023.09 | 74.47 | 80.8 | 56.71 | |
| DeCoGLMParameters=335M, Evaluation Protocol=Primary2024.05 | 74.4 | 77.4 | 64.6 | |
| TemplateGECParameters=125M+770M, Evaluation Protocol=Primary2024.05 | 74.1 | 76.8 | 64.8 | |
| GECTORBackbone=ROBERTa + XLNet, Ensemble=true2020.05 | 73.7 | 79.4 | 57.2 | |
| Omelianchuk et al.model_type=ensemble2022.03 | 73.7 | 79.4 | 57.2 | |
| Omelianchuk et al. (2020)Synthetic data=true, Model composition=Ensemble2023.07 | 73.7 | 79.4 | 57.2 | |
| Omelianchuk et al.2023.09 | 73.7 | 79.4 | 57.2 | |
| GECTORBackbone=BERT + ROBERTa + XLNet, Ensemble=true2020.05 | 73.6 | 78.9 | 58.2 | |
| DeGLM-CoGLMParameters=335M+335M, Evaluation Protocol=Primary2024.05 | 73.4 | 76.4 | 63.4 | |
| GECToR-Large (RoBERTa)model_type=single model, backbone=RoBERTa-Large, training_strategy=multi-stage training2022.03 | 73.21 | 80.7 | 53.39 | |
| GECTOR-LargeBackbone=RoBERTa, Configuration=Base system replacement for model [3]2023.10 | 73.21 | 80.7 | 53.39 | |
| Kantor et al. (2019)Ensemble=true2020.05 | 73.2 | 78.3 | 58 | |
| Kantor et al. (2019)Ensemble=true2021.09 | 73.2 | 78.3 | 58 | |
| GEC System Combination (1+2)combination_of=UEDIN-MS, Kakao&Brain2019.06 | 73.18 | 78.31 | 58 | |
| Lichtarge et al.2023.09 | 73 | 75.4 | 64.7 | |
| + BIFI (ours)Ensemble=false2021.09 | 72.9 | 79.4 | 55 | |
| Sun et al. (2021)Extra Data Size=300M, Transformer Config=12+2,1024,4096, PLM=true, syntax=false2022.10 | 72.9 | — | — | |
| SynGECExtra Data Size=2.4M, Transformer Config=12+12,1024,4096, PLM=true, syntax=true2022.10 | 72.9 | 75.1 | 65.5 | |
| SynGECParameters=110M+400M, Evaluation Protocol=Primary2024.05 | 72.9 | 75.1 | 65.5 | |
| GEC System Combination (1+2+3+4)combination_of=UEDIN-MS, Kakao&Brain, Shuyao, CAMB-CUED2019.06 | 72.84 | 78.74 | 56.04 | |
| GECToR-Large (RoBERTa@5K)model_type=single model, backbone=RoBERTa-Large, training_strategy=one-stage training2022.03 | 72.69 | 80.55 | 52.27 | |
| T5-Large2023.10 | 72.66 | 74.3 | 66.75 | |
| GECTORBackbone=XLNet, Ensemble=false2020.05 | 72.4 | 79.2 | 53.9 | |
| GECTOR (Omelianchuk et al., 2020)Ensemble=false2021.09 | 72.4 | 79.2 | 53.9 | |
| GECTOR (our base)Ensemble=false2021.09 | 72.4 | 79.2 | 53.9 | |
| Omelianchuk et al.model_type=single model2022.03 | 72.4 | 79.2 | 53.9 | |
| Omelianchuk et al. (2020)Extra Data Size=9M, Transformer Config=12+0,768,3072, PLM=true, syntax=false2022.10 | 72.4 | 79.2 | 53.9 | |
| Omelianchuk et al. (2020)Synthetic data=true, Model composition=Single2023.07 | 72.4 | 79.2 | 53.9 | |
| GECTOR XLNet2023.10 | 72.4 | 79.2 | 53.9 | |
| GECToRParameters=110M, Evaluation Protocol=Primary2024.05 | 72.4 | 79.2 | 53.9 | |
| GOParExtra Data Size=2.4M, Transformer Config=12+12,1024,4096, parser=Off-the-shelf Parser, PLM=true, syntax=true2022.10 | 72.3 | 74.6 | 64.1 | |
| Rothe et al. (2021)Extra Data Size=2.4M, Transformer Config=12+12,1024,4096, PLM=true, syntax=false2022.10 | 72.1 | — | — | |
| T5Parameters=770M, Evaluation Protocol=Primary2024.05 | 72.1 | — | — | |
| Our BaselineExtra Data Size=2.4M, Transformer Config=12+12,1024,4096, PLM=true, syntax=false2022.10 | 72 | 74 | 64.9 | |
| BARTParameters=400M, Evaluation Protocol=Primary2024.05 | 72 | 74 | 64.9 | |
| DeGLM-CoGLMParameters=335M+10B, Evaluation Protocol=LoRA Fine-tuning2024.05 | 71.69 | 72.8 | 67.57 | |
| GECTORBackbone=ROBERTa, Ensemble=false2020.05 | 71.5 | 77.2 | 55.1 | |
| GECTOR ROBERTa2023.10 | 71.5 | 77.2 | 55.1 | |
| Multi-EncoderParameters=110M+107M, Evaluation Protocol=Primary2024.05 | 70.6 | 73.3 | 61.5 | |
| Stahlberg and Kumarmodel_type=single model2022.03 | 70.4 | 72.1 | 64.4 | |
| Stahlberg and Kumar (2021)Extra Data Size=540M, Transformer Config=12+12,1024,4096, PLM=false, syntax=false2022.10 | 70.4 | 72.1 | 64.4 | |
| PRETLARGE+SSE+R2L+SEDEnsemble=true2019.09 | 70.2 | 74.7 | 56.7 | |
| Kiyono et al. (2019)Ensemble=true2020.05 | 70.2 | 74.7 | 56.7 | |
| Kiyono et al.*pseudo-data=true, ensemble=true2020.05 | 70.2 | 74.7 | 56.7 | |
| Kiyono et al.Configuration=Ensemble2020.05 | 70.2 | 74.7 | 56.7 | |
| Kiyono et al. (2019)Ensemble=true2021.09 | 70.2 | 74.7 | 56.7 | |
| Kiyono et al.model_type=ensemble2022.03 | 70.2 | 74.7 | 56.7 | |
| Kiyono et al. (2019)Synthetic data=true, Model composition=Ensemble2023.07 | 70.2 | 74.7 | 56.7 | |
| Riken&Tohoku2023.10 | 70.2 | 74.7 | 56.7 | |
| PRETLARGE+SSE+R2LEnsemble=true2019.09 | 69.8 | 72.1 | 61.8 | |
| BERT-fuse GED + R2Lpseudo-data=true, ensemble=true, R2L=true2020.05 | 69.8 | 72.3 | 61.4 | |
| Kaneko et al.Configuration=Ensemble2020.05 | 69.8 | 72.3 | 61.4 | |
| Kaneko et al.model_type=ensemble2022.03 | 69.8 | 72.3 | 61.4 | |
| GEC System Combination (3+4)combination_of=Shuyao, CAMB-CUED2019.06 | 69.72 | 74.99 | 54.41 | |
| VERNet2023.10 | 69.67 | 73.19 | 58.42 | |
| Grundkiewicz et al. (2019)Ensemble=true2019.09 | 69.5 | 72.3 | 60.1 | |
| Grundkiewicz et al.pseudo-data=true, ensemble=true2020.05 | 69.5 | 72.3 | 60.1 | |
| Grundkiewicz et al. (2019)Ensemble=true2021.09 | 69.5 | 72.3 | 60.1 | |
| Grundkiewicz et al.model_type=ensemble2022.03 | 69.5 | 72.3 | 60.1 | |
| UEDIN-MStype=standalone2019.06 | 69.47 | 72.28 | 60.12 | |
| UEDIN-MS2023.10 | 69.47 | 72.28 | 60.12 | |
| LLaMA2Parameters=13B, Evaluation Protocol=LoRA Fine-tuning2024.05 | 69.42 | 69.46 | 69.28 | |
| AMR-GECSynthetic data=false, Model composition=Ensemble2023.07 | 69.1 | 73.5 | 55.9 | |
| Kakao&Braintype=standalone2019.06 | 69 | 75.19 | 51.91 | |
| Kakao&Brain2023.10 | 69 | 75.19 | 51.91 | |
| SynGECExtra Data Size=2.4M, Transformer Config=6+6,512,2048, PLM=false, syntax=true2022.10 | 68.4 | 70.9 | 59.9 | |
| AMR-GECSynthetic data=false, Model composition=Single2023.07 | 68.4 | 71.5 | 58.3 | |
| GEC-DePenDParameters=253M, Evaluation Protocol=Primary2024.05 | 67.9 | 72.9 | 53.2 | |
| GECTORBackbone=BERT, Ensemble=false2020.05 | 67.6 | 71.5 | 55.7 | |
| GPT-22023.10 | 67.38 | 67.2 | 68.08 | |
| Wan and Wan (2021)Extra Data Size=10M, Transformer Config=6+6,512,2048, PLM=false, syntax=true2022.10 | 67.3 | 74.5 | 48.6 | |
| Rothe et al. (2021)Synthetic data=false, Model composition=Single2023.07 | 67.1 | — | — | |
| SpanDCParameters=125M+209M, Evaluation Protocol=Primary2024.05 | 66.9 | 70.4 | 55.9 | |
| SOME2023.10 | 66.68 | 66.4 | 67.83 | |
| Shuyaotype=standalone2019.06 | 66.61 | 70.17 | 55.39 | |
| Lichtarge et al. (2020)Extra Data Size=340M, Transformer Config=12+12,1024,4096, PLM=false, syntax=false2022.10 | 66.5 | 67.6 | 62.5 | |
| LLaMA2Parameters=7B, Evaluation Protocol=LoRA Fine-tuning2024.05 | 66.15 | 66.16 | 66.12 | |
| BART-basedConfiguration=Ensemble2020.05 | 66.1 | 68.8 | 57.1 | |
| Katsumata and Komachi (2020)Synthetic data=false, Model composition=Ensemble2023.07 | 66.1 | 68.8 | 57.1 | |
| REP(SIM)+WDROPPos=both2021.04 | 65.97 | — | — | |
| REP(UNI)+WDROPPos=both2021.04 | 65.94 | — | — | |
| ADVPos=both2021.04 | 65.9 | — | — |