Dialogue State Tracking on WOZ 2.0 (test)
91.37Joint Goal AccuracyAG-DST
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AG-DST2021.10 | 91.37 | — | |
| Seq2Seq-DU2021.10 | 91.2 | — | |
| SUMBTHistory representation=full history2021.08 | 91 | — | |
| PegasusHistory representation=full history2021.08 | 91 | — | |
| T5 (span)History representation=full history2021.08 | 91 | — | |
| T5 (span+ARLM)History representation=full history2021.08 | 91 | — | |
| Full BERT-based modelNumber of parameters=110M, Storage Size=440MB, Inference Time on CPU=1.465s, Inference Time on GPU=0.113s2019.10 | 90.5 | 97.6 | |
| Distilled BERT-based modelNumber of parameters=14M, Storage Size=55MB, Inference Time on CPU=0.205s, Inference Time on GPU=0.024s2019.10 | 90.4 | 97.7 | |
| T5 (ARLM)History representation=full history2021.08 | 89.5 | — | |
| StateNet_PSI2018.10 | 88.9 | — | |
| StateNet_PSI2019.10 | 88.9 | — | |
| StateNet2019.10 | 88.9 | — | |
| StateNet2021.10 | 88.9 | — | |
| StateNet-PSIHistory representation=full history2021.08 | 88.9 | — | |
| G-SATembeddings=Pre-trained GloVe + character n-gram2019.10 | 88.7 | 96.9 | |
| COMER2021.10 | 88.6 | — | |
| GCE2018.12 | 88.51 | 97.38 | |
| GCE2019.10 | 88.5 | 97.4 | |
| GCE2019.10 | 88.5 | 97.4 | |
| GCEHistory representation=full history2021.08 | 88.5 | — | |
| StateNet_PS2018.10 | 88.2 | — | |
| GLAD2018.05 | 88.1 | 97.1 | |
| GLAD2018.10 | 88.1 | — | |
| GLAD2019.10 | 88.1 | 97.1 | |
| GLADNumber of parameters=17M2019.10 | 88.1 | 97.1 | |
| GLAD2021.10 | 88.1 | — | |
| GLADHistory representation=full history2021.08 | 88.1 | — | |
| StateNet2018.10 | 87.8 | — | |
| BERT-DST PS2019.10 | 87.7 | — | |
| BERT-DST2021.10 | 87.7 | — | |
| CNN2019.10 | 86.9 | 95.4 | |
| Belief Tracking: CNN2018.10 | 85.5 | — | |
| Belief Tracking2021.10 | 85.5 | — | |
| Belief Tracking: Bi-LSTM2018.10 | 85.1 | — | |
| NBT-DNNArchitecture=Neural Belief Tracker, Word Vectors=Paragram-SL9992016.06 | 84.4 | 91.2 | |
| Neural Belief Tracker (NBT)architecture=DNN2018.05 | 84.4 | 91.2 | |
| Neural Belief Tracker: NBT-DNN2018.10 | 84.4 | — | |
| Neural Belief TrackerArchitecture=DNN2018.12 | 84.4 | 91.2 | |
| NBT - DNN2019.10 | 84.4 | 91.2 | |
| Neural Belief Tracker - DNN2019.10 | 84.4 | 91.2 | |
| Neural Belief Tracker: NBT-DNNHistory representation=full history2021.08 | 84.4 | — | |
| NBT-CNNArchitecture=Neural Belief Tracker, Word Vectors=Paragram-SL9992016.06 | 84.2 | 91.6 | |
| Neural Belief Tracker (NBT)architecture=CNN2018.05 | 84.2 | 91.6 | |
| Neural Belief Tracker: NBT-CNN2018.10 | 84.2 | — | |
| Neural Belief TrackerArchitecture=CNN2018.12 | 84.2 | 91.6 | |
| NBT - CNN2019.10 | 84.2 | 91.6 | |
| Neural Belief Tracker - CNN2019.10 | 84.2 | 91.6 | |
| Neural Belief Tracker2021.10 | 84.2 | — | |
| Neural Belief Tracker: NBT-CNNHistory representation=full history2021.08 | 84.2 | — | |
| Delexicalisation-Based Model + Semantic Dictionary2016.06 | 83.7 | 87.6 | |
| Delex. Model + Semantic Dictionarysemantic_dictionary=true2018.05 | 83.7 | 87.6 | |
| DB Model + Semantic Dictionary2018.10 | 83.7 | — | |
| Delex. ModelSemantic Dictionary=true2018.12 | 83.7 | 87.6 | |
| AUXGANWord Vectors=GLOVE-CC, DST Model=Neural Belief Tracker (NBT), Specialization Type=Post-Specialized2018.09 | 83.6 | — | |
| POST-DFFNWord Vectors=GLOVE-CC, DST Model=Neural Belief Tracker (NBT), Specialization Type=Post-Specialized2018.09 | 82.9 | — | |
| ATTRACT-REPELWord Vectors=GLOVE-CC, DST Model=Neural Belief Tracker (NBT), Specialization Type=Specialized2018.09 | 81.7 | — | |
| DistributionalWord Vectors=GLOVE-CC, DST Model=Neural Belief Tracker (NBT)2018.09 | 79.7 | — | |
| Delexicalisation-Based Model2016.06 | 70.8 | 87.1 | |
| Delexicalisation-Based Model2018.05 | 70.8 | 87.1 | |
| Delexicalisation-Based (DB) Model2018.10 | 70.8 | — | |
| Delex. Model2018.12 | 70.8 | 87.1 | |
| Delexicalisation Model2019.10 | 70.8 | 87.1 | |
| AUXGANVector space=FASTTEXT, Specialization protocol=Zero-shot2018.09 | 65.1 | — | |
| No pre-trainHistory representation=full history2021.08 | 64.5 | — | |
| DistributionalVector space=FASTTEXT, Specialization protocol=None2018.09 | 62.1 | — |