Response Generation on MultiWOZ (test)
35.1BLEU ScoreT5-Base
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| T5-Basemodel size=Base, beam width=102020.05 | 35.1 | — | — | — | — | — | — | — | — | 0.99 | |
| T5-3Bmodel size=3B, beam width=102020.05 | 34.8 | — | — | — | — | — | — | — | — | 0.86 | |
| T5-Largemodel size=Large, beam width=102020.05 | 34.7 | — | — | — | — | — | — | — | — | 0.92 | |
| Naive2020.04 | 34.6 | — | — | — | — | — | — | — | — | 1.27 | |
| T5-Smallmodel size=Small, beam width=102020.05 | 34.6 | — | — | — | — | — | — | — | — | 1.27 | |
| T2G22020.04 | 34.4 | — | — | — | — | — | — | — | — | 1.85 | |
| Schema2020.04 | 33.3 | — | — | — | — | — | — | — | — | 1.89 | |
| SC-GPT2020.04 | 30.8 | — | — | — | — | — | — | — | — | 0.53 | |
| SC-GPT2number of parameters=345M2020.05 | 30.8 | — | — | — | — | — | — | — | — | 0.53 | |
| SC-GPT2020.02 | 30.76 | — | — | — | 88.37 | — | — | — | — | — | |
| GPT-22020.02 | 30.71 | — | — | — | 87.7 | — | — | — | — | — | |
| HDSABelief State Type=oracle, System Action Type=oracle, System Action Form=graph2019.11 | 30.4 | — | 87.9 | 78 | — | — | 113.4 | — | — | — | |
| SFN + RLBelief State Type=oracle, System Action Type=oracle, System Action Form=one-hot2019.11 | 29 | — | — | — | — | — | 106 | — | — | — | |
| DAMD + multi-action data augmentationBelief State Type=oracle, System Action Type=oracle, System Action Form=span2019.11 | 27.3 | — | 95.4 | 87.2 | — | — | 118.5 | — | — | — | |
| HDSA2020.04 | 26.5 | — | — | — | — | — | — | — | — | 12.14 | |
| HDSA2020.05 | 26.5 | — | — | — | — | — | — | — | — | 12.14 | |
| HDSA2020.02 | 26.48 | — | — | — | 87.3 | — | — | — | — | — | |
| HDSABelief State Type=oracle, System Action Type=generated, System Action Form=graph2019.11 | 23.6 | — | 82.9 | 68.9 | — | — | 99.5 | — | — | — | |
| SC-LSTM2020.02 | 21.6 | — | — | — | 80.42 | — | — | — | — | — | |
| Seq2Seq + AttentionBelief State Type=oracle2019.11 | 18.9 | — | 71.3 | 61 | — | — | 85.1 | — | — | — | |
| DAMD + multi-action data augmentationBelief State Type=oracle, System Action Type=generated, System Action Form=span2019.11 | 18.6 | — | 89.2 | 77.9 | — | — | 102.2 | — | — | — | |
| DAMDBelief State Type=oracle, System Action Type=generated, System Action Form=span2019.11 | 18.3 | — | 89.5 | 75.8 | — | — | 100.9 | — | — | — | |
| SFN + RLBelief State Type=generated, System Action Type=generated, System Action Form=one-hot2019.11 | 16.9 | — | 73.8 | 58.6 | — | — | 83 | — | — | — | |
| MD-SequicityBelief State Type=oracle2019.11 | 16.8 | — | 86.6 | 71.6 | — | — | 95.9 | — | — | — | |
| DAMD + multi-action data augmentationBelief State Type=generated, System Action Type=generated, System Action Form=span2019.11 | 16.6 | — | 76.3 | 60.4 | — | — | 85 | — | — | — | |
| SFN + RLBelief State Type=oracle, System Action Type=generated, System Action Form=one-hot2019.11 | 16.3 | — | 82.7 | 72.1 | — | — | 93.7 | — | — | — | |
| Seq2Seq + CopyBelief State Type=oracle2019.11 | 15.7 | — | 86.2 | 72 | — | — | 94.8 | — | — | — | |
| 3-layer TransformerDialog-Act=None, Layers=32019.05 | — | 19.1 | 71.1 | 59.9 | 55.1 | 15.2 | — | — | — | — | |
| 3-layer Transformer-inDialog-Act=Tree Act, Layers=32019.05 | — | 20.2 | 73.8 | 62.1 | 57.3 | 16.2 | — | — | — | — | |
| 3-layer Transformer-inDialog-Act=Graph Act (Predicted), Layers=32019.05 | — | 22.7 | 80.4 | 65.1 | 64.6 | 19.9 | — | — | — | — | |
| 3-layer Transformer-inDialog-Act=Graph Act (Groundtruth), Layers=32019.05 | — | 29.1 | 85.5 | 72.6 | 83.8 | 25.1 | — | — | — | — | |
| 3-layer Transformer-outDialog-Act=Tree Act, Layers=32019.05 | — | 19.9 | 74.4 | 61.1 | 57.4 | 16 | — | — | — | — | |
| 3-layer Transformer-outDialog-Act=Graph Act (Predicted), Layers=32019.05 | — | 22.5 | 80.8 | 64.8 | 64.2 | 19.3 | — | — | — | — | |
| HDSADialog-Act=Graph Act (Predicted), Layers=2, Heads=7/27, Components=+ SA2019.05 | — | 23.2 | 82.9 | 69.1 | 65.1 | 20.3 | — | — | — | — | |
| HDSADialog-Act=Graph Act (Predicted), Layers=3, Heads=10/7/272019.05 | — | 23.6 | 82.9 | 68.9 | 65.7 | 20.6 | — | — | — | — | |
| HDSADialog-Act=Graph Act (Groundtruth), Layers=3, Heads=10/7/272019.05 | — | 30.4 | 87.9 | 78 | 86.2 | 26.2 | — | — | — | — | |
| HDSA2020.02 | — | — | — | — | — | — | — | 2.34 | 2.42 | — | |
| Human2020.02 | — | — | — | — | — | — | — | 2.77 | 2.61 | — | |
| LSTMDialog-Act=None2019.05 | — | 18.8 | 71.2 | 60.2 | 54.8 | 15.1 | — | — | — | — | |
| SC-GPT2020.02 | — | — | — | — | — | — | — | 2.71 | 2.69 | — | |
| SC-LSTMDialog-Act=Tree Act2019.05 | — | 20.5 | 74.5 | 62.5 | 57.7 | 16.6 | — | — | — | — | |
| SC-LSTM2020.02 | — | — | — | — | — | — | — | 2.14 | 2.33 | — | |
| Straight DSADialog-Act=Graph Act (Predicted), Heads=44, Components=+ 2 x SA2019.05 | — | 22.6 | 80.3 | 67.1 | 65 | 20 | — | — | — | — | |
| Straight DSADialog-Act=Graph Act (Groundtruth), Heads=44, Components=+ 2 x SA2019.05 | — | 29.6 | 86.4 | 75.6 | 84.1 | 25.5 | — | — | — | — |