NL-to-code generation on CoNaLa (test)
35.32BLEUBART W/Mined
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BART W/MinedInput Context=With Body2021.06 | 35.32 | — | |
| BARTInput Context=With Body2021.06 | 34.35 | — | |
| TranX (Our best)Data Strategy=Man+Mine+API, method=dist., retrieval_target=code, reranking=true2020.04 | 32.26 | — | |
| EK+RRInput Context=No Body2021.06 | 32.26 | — | |
| TranXData Strategy=Man+Mine(100k), reranking=true2020.04 | 31.42 | — | |
| TranXData Strategy=Man+Mine+API, method=dist., retrieval_target=code, reranking=false2020.04 | 30.69 | — | |
| EKInput Context=No Body2021.06 | 30.69 | — | |
| BART W/MinedInput Context=No Body2021.06 | 30.55 | — | |
| TranXData Strategy=Man+Mine+API, method=direct, retrieval_target=code, reranking=false2020.04 | 30.26 | — | |
| TranXData Strategy=Man, reranking=true2020.04 | 30.11 | — | |
| RRInput Context=No Body2021.06 | 30.11 | — | |
| TranXData Strategy=Man+Mine+API, method=direct, retrieval_target=intent, reranking=false2020.04 | 29.66 | — | |
| TranXData Strategy=Man+Mine+API, method=dist., retrieval_target=intent, reranking=false2020.04 | 29.31 | — | |
| TranXData Strategy=Man+Mine, mined_data_size=100k, reranking=false2020.04 | 28.14 | — | |
| TranXData Strategy=Man+Mine, mined_data_size=50k, reranking=false2020.04 | 27.94 | — | |
| TranXData Strategy=Man+Mine+API, method=w/o re-sampling, reranking=false2020.04 | 27.84 | — | |
| TranXData Strategy=Man, reranking=false2020.04 | 27.2 | — | |
| BARTInput Context=No Body2021.06 | 26.24 | — | |
| TranXInput Context=No Body2021.06 | 24.3 | — | |
| EK+RRInput Context=With Body2021.06 | 20.54 | 33.71 | |
| EKInput Context=With Body2021.06 | 20.37 | 33.71 | |
| RRInput Context=With Body2021.06 | 19.85 | 31.21 | |
| TranXInput Context=With Body2021.06 | 18.85 | 31.21 |