Question Generation on SQuAD 1.1 (test)
25.8BLEU-4ProphetNet
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ProphetNetsplit_type=Zhao et al., epochs=10, learning rate=1e-5, batch size=282020.01 | 25.8 | — | — | — | 27.54 | 53.65 | — | — | — | — | — | — | |
| JGR-RDescription=using the ranker of JGR to re-rank the outputs of JGR-G2022.06 | 24.73 | — | — | — | 26.97 | 53.57 | — | — | — | — | — | — | |
| JGR-GDescription=generator model in JGR2022.06 | 24.52 | — | — | — | 26.46 | 52.79 | — | — | — | — | — | — | |
| ProphetNetsplit_type=Du et al., epochs=10, learning rate=1e-5, batch size=282020.01 | 23.91 | — | — | — | 26.6 | 52.26 | — | — | — | — | — | — | |
| UNILM2022.06 | 23.75 | — | — | — | 25.61 | 52.04 | — | — | — | — | — | — | |
| MASS2022.06 | 23.14 | — | — | — | 25.36 | 50.98 | — | — | — | — | — | — | |
| BARTBackbone=BART-large, Is backbone for JGR-G=true2022.06 | 23.14 | — | — | — | 26.56 | 51.46 | — | — | — | — | — | — | |
| UniLMsplit_type=Zhao et al.2020.01 | 23.08 | — | — | — | 25.57 | 52.03 | — | — | — | — | — | — | |
| ProphetNet2022.06 | 22.5 | — | — | — | 26 | 51.5 | — | — | — | — | — | — | |
| UniLMsplit_type=Du et al.2020.01 | 21.63 | — | — | — | 25.04 | 51.09 | — | — | — | — | — | — | |
| SemQGsplit_type=Zhao et al.2020.01 | 20.76 | — | — | — | 24.2 | 48.91 | — | — | — | — | — | — | |
| FreIns*Approach=Frequent Name with Insensitivity Training2023.05 | 19.71 | — | — | — | — | 57.29 | 10.09 | 3.12 | 1.35 | 4.48 | 4.19 | 1.8 | |
| IDApproach=In-distribution Baseline2023.05 | 19.21 | — | — | — | — | 56.49 | — | — | — | — | — | — | |
| FreApproach=Frequent Name Fine-tuning2023.05 | 18.96 | — | — | — | — | 57.1 | 18.44 | 5.51 | 2.35 | 8.35 | 7.23 | 3.04 | |
| FreAugApproach=Frequent Name Data Augmentation2023.05 | 18.52 | — | — | — | — | 57.06 | 16.01 | 4.92 | 2.14 | 7.05 | 6.5 | 2.76 | |
| SemQGsplit_type=Du et al.2020.01 | 18.37 | — | — | — | 22.65 | 46.68 | — | — | — | — | — | — | |
| MP-GSNsplit_type=Zhao et al.2020.01 | 16.38 | — | — | — | 20.25 | 44.48 | — | — | — | — | — | — | |
| CorefNQGsplit_type=Du et al.2020.01 | 15.16 | — | — | — | 19.12 | — | — | — | — | — | — | — | |
| NQG++Decoding=Beam search, Copy mechanism=true, Pre-trained GloVe=true, Shared embedding matrix=true2017.04 | 13.29 | — | — | — | — | — | — | — | — | — | — | — | |
| NQG+STshareDecoding=Beam search, Copy mechanism=true, Shared embedding matrix=true2017.04 | 12.8 | — | — | — | — | — | — | — | — | — | — | — | |
| NQG+PretrainDecoding=Beam search, Copy mechanism=true, Pre-trained GloVe=true2017.04 | 12.69 | — | — | — | — | — | — | — | — | — | — | — | |
| NQG+Decoding=Beam search, Copy mechanism=true2017.04 | 12.18 | — | — | — | — | — | — | — | — | — | — | — | |
| NQGDecoding=Beam search, Feature-rich encoder=true2017.04 | 10.13 | — | — | — | — | — | — | — | — | — | — | — | |
| NQG-CaseDecoding=Beam search, Word case feature=removed2017.04 | 9.89 | — | — | — | — | — | — | — | — | — | — | — | |
| NQG-POSDecoding=Beam search, POS tag feature=removed2017.04 | 9.87 | — | — | — | — | — | — | — | — | — | — | — | |
| PCFG-TransDecoding=Beam search, Type=Rule-based system2017.04 | 9.31 | — | — | — | — | — | — | — | — | — | — | — | |
| NQG-NERDecoding=Beam search, NER feature=removed2017.04 | 9.29 | — | — | — | — | — | — | — | — | — | — | — | |
| s2s+attDecoding=Beam search, Architecture=Seq2seq with attention2017.04 | 3.06 | — | — | — | — | — | — | — | — | — | — | — | |
| NQG-AnswerDecoding=Beam search, Answer position indicator=removed2017.04 | 2.98 | — | — | — | — | — | — | — | — | — | — | — | |
| Beam SearchK (number of questions)=3, Decoding strategy=Standard Beam Search2019.09 | — | 13.59 | 16.848 | 67.277 | — | — | — | — | — | — | — | — | |
| Beam SearchK (number of questions)=5, Decoding strategy=Standard Beam Search2019.09 | — | 13.526 | 18.809 | 74.674 | — | — | — | — | — | — | — | — | |
| Beam Search + Focus GuideK (number of questions)=5, Focus Guide during Test Time=true2019.09 | — | 24.58 | — | — | — | — | — | — | — | — | — | — | |
| Diverse Beam SearchK (number of questions)=3, Decoding strategy=Diverse Beam Search2019.09 | — | 13.696 | 16.989 | 68.018 | — | — | — | — | — | — | — | — | |
| Diverse Beam SearchK (number of questions)=5, Decoding strategy=Diverse Beam Search2019.09 | — | 13.379 | 18.298 | 74.795 | — | — | — | — | — | — | — | — | |
| Mixture DecoderK (number of mixtures)=3, Decoding strategy=Greedy2019.09 | — | 14.72 | 19.324 | 51.36 | — | — | — | — | — | — | — | — | |
| Mixture DecoderK (number of mixtures)=5, Decoding strategy=Greedy2019.09 | — | 15.166 | 21.965 | 58.727 | — | — | — | — | — | — | — | — | |
| NQG++2019.09 | — | 13.27 | — | — | — | — | — | — | — | — | — | — | |
| SELECTORK (number of mixtures)=3, Decoding strategy=Greedy2019.09 | — | 15.874 | 20.437 | 47.493 | — | — | — | — | — | — | — | — | |
| SELECTORK (number of mixtures)=5, Decoding strategy=Greedy2019.09 | — | 15.672 | 22.451 | 59.815 | — | — | — | — | — | — | — | — | |
| Truncated SamplingK (number of questions)=3, Decoding strategy=Truncated Sampling2019.09 | — | 11.89 | 15.447 | 37.372 | — | — | — | — | — | — | — | — | |
| Truncated SamplingK (number of questions)=5, Decoding strategy=Truncated Sampling2019.09 | — | 11.53 | 17.651 | 45.99 | — | — | — | — | — | — | — | — |