Task-Oriented Dialogue on CamRest676 (test)
87.1Task Success RateARDM
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ARDMPre-training=N/A, Decoding method=seq2seq, Number of parameters=N/A2021.03 | 87.1 | — | 25.2 | — | |
| SOLOISTPre-training=Task-Oriented, Decoding method=seq2seq, Number of parameters=N/A2021.03 | 87.1 | 94.7 | 25.5 | 116.4 | |
| ARDMBelief State Annotation=false, Policy Annotation=true2020.05 | 87.1 | — | 25.2 | — | |
| SOLOISTBelief State Annotation=true, Policy Annotation=false2020.05 | 87.1 | 94.7 | 25.5 | 116.4 | |
| GPT-2 fine-tunedPre-training=GPT-2, Decoding method=seq2seq, Number of parameters=N/A2021.03 | 86.2 | — | 19.2 | — | |
| GPT fine-tuningBelief State Annotation=false, Policy Annotation=false2020.05 | 86.2 | — | 19.2 | — | |
| SequicityPre-training=None, Decoding method=seq2seq, Number of parameters=N/A2021.03 | 85.3 | 92.3 | 21.4 | 110.2 | |
| Noisy channel online decodingPre-training=Task-Oriented Pretraining, Decoding method=Noisy channel online, Number of parameters=292M2021.03 | 85.3 | 95.4 | 26.89 | 117.24 | |
| SequicityBelief State Annotation=true, Policy Annotation=true2020.05 | 85.3 | 92.3 | 21.4 | 110.2 | |
| Noisy channel online decodingPre-training=Task-Oriented Pretraining, Decoding method=Noisy channel online, Number of parameters=116M2021.03 | 85.2 | 94.3 | 25.98 | 115.73 | |
| Noisy channel online decodingPre-training=Randomly Initialized, Decoding method=Noisy channel online, Number of parameters=292M2021.03 | 84.9 | 80.9 | 23.19 | 106.09 | |
| Noisy channel online decodingPre-training=Reddit Pretraining, Decoding method=Noisy channel online, Number of parameters=292M2021.03 | 84.7 | 93.9 | 25.38 | 114.68 | |
| Noisy channel online decodingPre-training=Reddit Pretraining, Decoding method=Noisy channel online, Number of parameters=116M2021.03 | 84.5 | 93.7 | 25.14 | 114.24 | |
| Direct decodingPre-training=Task-Oriented Pretraining, Decoding method=Direct, Number of parameters=114M2021.03 | 84.3 | 93.4 | 24.92 | 113.77 | |
| Noisy channel online decodingPre-training=Randomly Initialized, Decoding method=Noisy channel online, Number of parameters=116M2021.03 | 84.1 | 79.8 | 22.83 | 104.78 | |
| Direct decodingPre-training=Reddit Pretraining, Decoding method=Direct, Number of parameters=114M2021.03 | 83.9 | 93.3 | 23.41 | 112.01 | |
| Direct decodingPre-training=Randomly Initialized, Decoding method=Direct, Number of parameters=114M2021.03 | 83.5 | 78.1 | 21.58 | 102.38 | |
| Sequicity (w/o RL)Belief State Annotation=true, Policy Annotation=true2020.05 | 83.4 | 94 | 23.4 | 112.1 |