End-to-end task-oriented dialogue on MultiWOZ (test)
94Task Success RateEWC+RL
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EWC+RLBelief State=Oracle2021.07 | 94 | — | — | — | 100.2 | 103 | 93.9 | 82.6 | 95 | 89.2 | — | — | — | — | — | |
| Naive+RLBelief State=Oracle2021.07 | 93.9 | — | — | — | 99.7 | 104.3 | 92 | 80.6 | 97.2 | 89.3 | — | — | — | — | — | |
| JOUST RL-Joint w/ turn-RBelief State=Oracle, Reward type=Turn-level2021.07 | 86.7 | 94.7 | 18.7 | 109.4 | — | — | — | — | — | — | — | — | — | — | — | |
| POKI2022.03 | 86 | 84 | — | — | — | — | — | — | — | — | 43 | 76 | — | — | — | |
| JOUST RL-Joint w/ dial-RBelief State=Oracle, Reward type=Dialogue-level2021.07 | 85.7 | 93.9 | 16.9 | 106.7 | — | — | — | — | — | — | — | — | — | — | — | |
| EWCBelief State=Oracle2021.07 | 85.5 | — | — | — | 95.5 | 96.9 | 89.6 | 70 | 81.5 | 79.6 | — | — | — | — | — | |
| MORE2026.06 | 84.8 | 92.5 | 18.7 | 107.35 | — | — | — | — | — | — | — | — | 2.96 | 3,226 | 25,834 | |
| NaiveBelief State=Oracle2021.07 | 83.6 | — | — | — | 88.8 | 98.4 | 85.9 | 72.2 | 79.8 | 76.7 | — | — | — | — | — | |
| ProTOD2026.06 | 83.3 | 91.7 | 8.9 | 96.4 | — | — | — | — | — | — | — | — | 3.26 | 1,951 | 14,345 | |
| AutoTOD2026.06 | 82.8 | 87.2 | 9.3 | 94.3 | — | — | — | — | — | — | — | — | 2.62 | 1,722 | 10,188 | |
| RewardNet2026.06 | 81.5 | 87.6 | 17.6 | 102.2 | — | — | — | — | — | — | — | — | 1.99 | 423 | 3,942 | |
| TOATOD2026.06 | 79.8 | 90 | 17 | 101.9 | — | — | — | — | — | — | — | — | — | — | — | |
| UBAR2021.03 | 79.5 | 88.2 | 16.43 | 100.28 | — | — | — | — | — | — | — | — | — | — | — | |
| JOUST Supervised LearningBelief State=Oracle, Learning regime=Supervised Learning2021.07 | 79.4 | 88.5 | 18.3 | 102.3 | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOISTBelief State=Oracle, Pre-trained transformer-based=true2021.07 | 79.3 | 89.6 | 18.3 | 102.5 | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOISTAnnotations: Belief State=false, Annotations: Policy=false2020.05 | 79.3 | 89.6 | 18.03 | 102.49 | — | — | — | — | — | — | — | — | — | — | — | |
| LaRLAnnotations: Belief State=false, Annotations: Policy=true2020.05 | 79.2 | 82.8 | 12.8 | 93.8 | — | — | — | — | — | — | — | — | — | — | — | |
| PARGBelief State=Oracle, Pre-trained transformer-based=false2021.07 | 78.9 | 91.1 | 18.8 | 103.8 | — | — | — | — | — | — | — | — | — | — | — | |
| MarCoBelief State=Oracle, Pre-trained transformer-based=true2021.07 | 78.6 | 92.3 | 20 | 105.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Mars2026.06 | 78 | 88.9 | 19.9 | 103.4 | — | — | — | — | — | — | — | — | 1.65 | 288 | 2,264 | |
| DAMDBelief State=Oracle, Pre-trained transformer-based=false2021.07 | 77.9 | 89.2 | 18.6 | 102.2 | — | — | — | — | — | — | — | — | — | — | — | |
| DAMDAnnotations: Belief State=true, Annotations: Policy=true2020.05 | 77.9 | 89.2 | 18.6 | 102.15 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel online decodingPre-training strategy=Task-Oriented Pretraining, Number of parameters=292M2021.03 | 76.2 | 86.9 | 20.58 | 102.13 | — | — | — | — | — | — | — | — | — | — | — | |
| GALAXY2026.06 | 75.7 | 85.4 | 19.6 | 100.2 | — | — | — | — | — | — | — | — | 1.75 | 295 | 2,275 | |
| Noisy channel rerankingPre-training strategy=Task-Oriented Pretraining, Number of parameters=292M2021.03 | 74.9 | 86.5 | 20.31 | 101.01 | — | — | — | — | — | — | — | — | — | — | — | |
| MinTL-BARTBelief state=Predicted, Leverages pre-trained transformer=true2021.07 | 74.9 | 84.9 | 17.9 | 97.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel online decodingPre-training strategy=Task-Oriented Pretraining, Number of parameters=116M2021.03 | 74.8 | 85.9 | 19.76 | 100.11 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel rerankingPre-training strategy=Task-Oriented Pretraining, Number of parameters=116M2021.03 | 73.8 | 85.6 | 19.38 | 99.08 | — | — | — | — | — | — | — | — | — | — | — | |
| JOUST RL-Joint w/ turn-RBelief state=Predicted, Training Mode=RL-Joint, Reward=Turn-level (turn-R)2021.07 | 73.5 | 83.2 | 17.6 | 96 | — | — | — | — | — | — | — | — | — | — | — | |
| MoGNetBelief State=Oracle, Pre-trained transformer-based=false2021.07 | 73.3 | 85.3 | 20.1 | 99.4 | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOIST2021.03 | 72.9 | 85.5 | 16.54 | 95.74 | — | — | — | — | — | — | — | — | — | — | — | |
| Direct decodingPre-training strategy=Task-Oriented Pretraining, Number of parameters=114M2021.03 | 72.9 | 85.2 | 17 | 96.05 | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOISTBelief state=Predicted, Leverages pre-trained transformer=true2021.07 | 72.9 | 85.5 | 16.5 | 95.7 | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOISTBelief State=true, Policy=false2020.05 | 72.9 | 85.5 | 16.54 | 95.74 | — | — | — | — | — | — | — | — | — | — | — | |
| ARDMBelief State=Oracle, Pre-trained transformer-based=true2021.07 | 72.8 | 87.4 | 20.6 | 100.7 | — | — | — | — | — | — | — | — | — | — | — | |
| ARDMAnnotations: Belief State=false, Annotations: Policy=false2020.05 | 72.8 | 87.4 | 20.6 | 100.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Structured FusionAnnotations: Belief State=true, Annotations: Policy=false2020.05 | 72.1 | 82.7 | 16.34 | 93.74 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel online decodingPre-training strategy=Reddit Pretraining, Number of parameters=292M2021.03 | 71.7 | 82.4 | 20.49 | 97.54 | — | — | — | — | — | — | — | — | — | — | — | |
| MD-SequicityAnnotations: Belief State=true, Annotations: Policy=false2020.05 | 71.6 | 86.6 | 16.68 | 95.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel online decodingPre-training strategy=Reddit Pretraining, Number of parameters=116M2021.03 | 71.1 | 81.6 | 19.31 | 95.66 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel rerankingPre-training strategy=Reddit Pretraining, Number of parameters=292M2021.03 | 70.9 | 82.2 | 19.89 | 96.44 | — | — | — | — | — | — | — | — | — | — | — | |
| UBAR2026.06 | 70.3 | 83.4 | 17.6 | 94.5 | — | — | — | — | — | — | — | — | 2.1 | 478 | 5,238 | |
| SimpleTOD2021.03 | 70.1 | 84.4 | 15.01 | 92.26 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel rerankingPre-training strategy=Reddit Pretraining, Number of parameters=116M2021.03 | 70.1 | 81.3 | 19.01 | 94.71 | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTODBelief state=Predicted, Leverages pre-trained transformer=true2021.07 | 70.1 | 84.4 | 15 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTOD2026.06 | 70.1 | 84.4 | 15 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | |
| SGP-TOD2026.06 | 69.9 | 83.9 | 9.1 | 86 | — | — | — | — | — | — | — | — | — | — | — | |
| JOUST RL-Joint w/ dial-RBelief state=Predicted, Training Mode=RL-Joint, Reward=Dialogue-level (dial-R)2021.07 | 69.4 | 80.6 | 17.5 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Direct decodingPre-training strategy=Reddit Pretraining, Number of parameters=114M2021.03 | 69.2 | 81.8 | 17.06 | 92.16 | — | — | — | — | — | — | — | — | — | — | — | |
| Rewriter2022.03 | 69 | 35 | — | — | — | — | — | — | — | — | 82 | 56 | — | — | — | |
| HDSAAnnotations: Belief State=true, Annotations: Policy=true2020.05 | 68.9 | 82.9 | 23.6 | 99.5 | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTODBelief State=Oracle, Pre-trained transformer-based=true2021.07 | 67.1 | 88.9 | 16.9 | 94.9 | — | — | — | — | — | — | — | — | — | — | — | |
| JOUST Supervised LearningBelief state=Predicted, Training Mode=Supervised Learning2021.07 | 66.7 | 77.4 | 17.4 | 89.5 | — | — | — | — | — | — | — | — | — | — | — | |
| SourceBelief State=Oracle2021.07 | 64.4 | — | — | — | 82.3 | 93.3 | 76.2 | 36.8 | 55.4 | 42.4 | — | — | — | — | — | |
| DSTC8 Track 1 Winner2021.03 | 62.4 | 73 | 16 | 83.5 | — | — | — | — | — | — | — | — | — | — | — | |
| DSTC8 Track 1 WinnerBelief State=true, Policy=true2020.05 | 62.4 | 73 | 16 | 83.5 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT fine-tuningAnnotations: Belief State=true, Annotations: Policy=false2020.05 | 61.36 | 70.96 | 19.05 | 85.21 | — | — | — | — | — | — | — | — | — | — | — | |
| BaselineAnnotations: Belief State=false, Annotations: Policy=false2020.05 | 60.94 | 71.29 | 18.8 | 84.93 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel online decodingPre-training strategy=Randomly Initialized, Number of parameters=292M2021.03 | 60.9 | 83.9 | 15.57 | 87.97 | — | — | — | — | — | — | — | — | — | — | — | |
| DAMD2021.03 | 60.4 | 76.4 | 16.6 | 85 | — | — | — | — | — | — | — | — | — | — | — | |
| DAMDBelief state=Predicted, Leverages pre-trained transformer=false2021.07 | 60.4 | 76.3 | 16.6 | 85 | — | — | — | — | — | — | — | — | — | — | — | |
| DAMDBelief State=true, Policy=true2020.05 | 60.4 | 76.4 | 16.6 | 85 | — | — | — | — | — | — | — | — | — | — | — | |
| TokenMoEAnnotations: Belief State=true, Annotations: Policy=false2020.05 | 59.7 | 75.3 | 16.81 | 84.31 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel online decodingPre-training strategy=Randomly Initialized, Number of parameters=116M2021.03 | 58.9 | 82.9 | 15.33 | 86.23 | — | — | — | — | — | — | — | — | — | — | — | |
| Structured FusionBelief State=true, Policy=true2020.05 | 58.6 | 73.8 | 16.9 | 83.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Noisy channel rerankingPre-training strategy=Randomly Initialized, Number of parameters=292M2021.03 | 58.1 | 82.1 | 15.37 | 85.47 | — | — | — | — | — | — | — | — | — | — | — | |
| HRED-TS2021.03 | 58 | 70 | 17.5 | 81.5 | — | — | — | — | — | — | — | — | — | — | — | |
| HRED-TSBelief state=Predicted, Leverages pre-trained transformer=false2021.07 | 58 | 70 | 17.5 | 81.5 | — | — | — | — | — | — | — | — | — | — | — | |
| EWC+RLBelief State=Predicted2021.07 | 58 | — | — | — | 64.7 | 77.6 | 67.6 | 46.6 | 43.2 | 48.5 | — | — | — | — | — | |
| HRED-TSBelief State=true, Policy=true2020.05 | 58 | 70 | 17.5 | 81.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Naive+RLBelief State=Predicted2021.07 | 57.5 | — | — | — | 63.2 | 74.4 | 68.4 | 47.4 | 42.7 | 48.7 | — | — | — | — | — | |
| Noisy channel rerankingPre-training strategy=Randomly Initialized, Number of parameters=116M2021.03 | 57.1 | 82.7 | 15.29 | 85.19 | — | — | — | — | — | — | — | — | — | — | — | |
| Direct decodingPre-training strategy=Randomly Initialized, Number of parameters=114M2021.03 | 54.7 | 81 | 15.12 | 82.97 | — | — | — | — | — | — | — | — | — | — | — | |
| EWCBelief State=Predicted2021.07 | 53.9 | — | — | — | 57.4 | 72.1 | 66.1 | 43.7 | 39 | 45 | — | — | — | — | — | |
| NaiveBelief State=Predicted2021.07 | 51.7 | — | — | — | 57.2 | 69.2 | 65 | 40.3 | 36 | 42.8 | — | — | — | — | — | |
| SequicityBelief State=true, Policy=true2020.05 | 45.32 | 66.41 | 15.54 | 71.41 | — | — | — | — | — | — | — | — | — | — | — | |
| Sequicity2021.03 | 45.3 | 66.4 | 15.54 | 71.39 | — | — | — | — | — | — | — | — | — | — | — | |
| SourceBelief State=Predicted2021.07 | 34 | — | — | — | 46 | 55.4 | 34.3 | 22 | 26.6 | 19.9 | — | — | — | — | — |