Dialogue State Tracking on MultiWOZ 2.1 (test)
60.76Joint Goal AccuracyESAinsTOD (GT)
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ESAinsTOD (GT)Approach category=Generation-based, Backbone=Llama 2 7B, Evaluation protocol=Ground Truth2026.03 | 60.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSS-DSTApproach category=Hybrid, Full ontology requirement=true2026.03 | 60.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Trippy+SaCLogBackbone Model=BERTbase, Trainable Parameters=110M2023.05 | 60.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPy + SaCLogPre-trained Model=BERT-base, Use Modified Label=true2022.11 | 60.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPy + CoCoAugPre-trained Model=BERT-base, Use Modified Label=true2022.11 | 60.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPy + SCoRePre-trained Model=BERT-base, Use Modified Label=true2022.11 | 60.48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPy + SCOREPretrained-Model=ROBERTA-large, Parameters=355M, extra dialogue data=true2021.09 | 60.48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FPDSC-dualApproach category=Classification-based, Full ontology requirement=true2026.03 | 59.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ConvBERT+DG+MultiBackbone Model=BERTbase, Trainable Parameters=110M2023.05 | 58.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ESAinsTODApproach category=Generation-based, Backbone=Llama 2 7B2026.03 | 58.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHANApproach category=Classification-based, Full ontology requirement=true2026.03 | 58.55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FPDSC-turnApproach category=Classification-based, Full ontology requirement=true2026.03 | 57.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| D3STXXLBackbone Model=T5XXL, Trainable Parameters=11B2023.05 | 57.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MoNETPre-trained Model=BERT-base2022.11 | 57.71 | — | — | — | — | — | — | 97.71 | — | — | — | — | — | — | — | — | — | |
| SPACE-3Backbone Model=UniLM, Trainable Parameters=340M2023.05 | 57.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PPTODlargeBackbone Model=T5large, Trainable Parameters=770M2023.05 | 57.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PPTODPre-trained Model=T5-large2022.11 | 57.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PPTODApproach category=Generation-based, Task-specific training (DST)=true, Dataset-specific fine-tuning=true2026.03 | 57.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AG-DSTBackbone Model=PLATO-2, Trainable Parameters=310M2023.05 | 57.26 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PPTODbaseBackbone Model=T5base, Trainable Parameters=220M2023.05 | 57.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOISTPretrained-Model=GPT-2, Parameters=117M, extra dialogue data=true2021.09 | 56.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOIST (DST)Approach category=Generation-based, Task-specific training (DST)=true2026.03 | 56.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 2 7BApproach category=Generation-based, Evaluation protocol=Fine-tuned2026.03 | 56.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSGFNetPre-trained Model=BERT-base2022.11 | 56.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SDP-DSTBackbone Model=T5base, Trainable Parameters=220M2023.05 | 56.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SDP-DSTPre-trained Model=T5-base2022.11 | 56.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Independent (ours)Pretrained-Model=T5-base, Parameters=220M, trained with description=true2021.09 | 56.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTODBackbone Model=DistilGPT-2, Trainable Parameters=82M2023.05 | 56.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Independent (ours)Pretrained-Model=T5-base, Parameters=220M, trained with description=false2021.09 | 56.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STARPre-trained Model=BERT-base2022.11 | 56.36 | — | — | — | — | — | — | 97.59 | — | — | — | — | — | — | — | — | — | |
| UBAR (DST)Approach category=Generation-based, Task-specific training (DST)=true2026.03 | 56.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Independent (ours)Pretrained-Model=T5-small, Parameters=60M, trained with description=true2021.09 | 56.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Seq2SeqDUPre-trained Model=BERT-base2022.11 | 56.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Seq2seq-DUApproach category=Generation-based2026.03 | 56.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaselinePre-trained Model=BERT-base2022.11 | 55.82 | — | — | — | — | — | — | 97.51 | — | — | — | — | — | — | — | — | — | |
| SimpleTODApproach category=Generation-based2026.03 | 55.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Simple-TODPretrained-Model=GPT2, Parameters=117M2021.09 | 55.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTODHistory=Full, Source=Original paper2021.08 | 55.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Independent (ours)Pretrained-Model=T5-small, Parameters=60M, trained with description=false2021.09 | 55.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-DSTHistory=Full2021.08 | 55.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPyBackbone Model=BERTbase, Trainable Parameters=110M2023.05 | 55.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPyHistory=Full2021.08 | 55.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPyPre-trained Model=BERT-base, Use Modified Label=true2022.11 | 55.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPyPretrained-Model=BERT-base, Parameters=110M2021.09 | 55.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripPyApproach category=Classification-based2026.03 | 55.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTODPre-trained Model=DistilGPT22022.11 | 55.26 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SSTPre-trained Model=BERT-base2022.11 | 55.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SSTPretrained-Model=N, Parameters=N/A2021.09 | 55.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SSTApproach category=Classification-based, Full ontology requirement=true2026.03 | 55.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TOATODbaseBackbone Model=T5base, Trainable Parameters=36M2023.05 | 54.97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SAVNPre-trained Model=BERT-base2022.11 | 54.86 | — | — | — | — | — | — | 97.55 | — | — | — | — | — | — | — | — | — | |
| D3STlargeBackbone Model=T5large, Trainable Parameters=770M2023.05 | 54.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SAVNHistory=Full2021.08 | 54.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PegasusHistory=Full2021.08 | 54.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniLMBackbone Model=UniLM, Trainable Parameters=340M2023.05 | 54.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| D3STbaseBackbone Model=T5base, Trainable Parameters=220M2023.05 | 54.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MDTOD2023.05 | 53.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MinTLPretrained-Model=BART-large, Parameters=406M2021.09 | 53.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MinTLApproach category=Generation-based2026.03 | 53.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MTTOD2023.05 | 53.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOLOIST2023.05 | 53.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DST-PicklistBackbone Model=BERTbase, Trainable Parameters=110M2023.05 | 53.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DST-PicklistPre-trained Model=BERT-base2022.11 | 53.3 | — | — | — | — | — | — | 97.4 | — | — | — | — | — | — | — | — | — | |
| DST-PicklistApproach category=Classification-based, Full ontology requirement=true2026.03 | 53.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CSFN-DSTPre-trained Model=BERT-base2022.11 | 53.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5 (span+ARLM)History=Full2021.08 | 53.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TOATODsmallBackbone Model=T5small, Trainable Parameters=7.9M2023.05 | 53.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOM-DSTPre-trained Model=BERT-base2022.11 | 53.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5 (span)History=Full2021.08 | 52.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOM-DST2023.05 | 52.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOM-DSTApproach category=Generation-based2026.03 | 52.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5 (ARLM)History=Full2021.08 | 52.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5-Basebackbone=T5-Base2023.05 | 52.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PPTODevaluation_corrected=true2023.05 | 51.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DS-DSTApproach category=Hybrid2026.03 | 51.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DS-DSTBackbone Model=BERTbase, Trainable Parameters=110M2023.05 | 51.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DS-DSTHistory=Full2021.08 | 51.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOM-DSTHistory=Full2021.08 | 51.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSTQA2023.05 | 51.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MinTLPretrained-Model=T5-small, Parameters=60M2021.09 | 50.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTODHistory=Full, Source=2.3 author2021.08 | 50.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimpleTODevaluation_corrected=true2023.05 | 50.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| dual-DST2023.05 | 49.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSTQAApproach category=Hybrid, Full ontology requirement=true2026.03 | 49.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SUMBTHistory=Full2021.08 | 49.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SUMBTPre-trained Model=BERT-base2022.11 | 49.01 | — | — | — | — | — | — | 96.76 | — | — | — | — | — | — | — | — | — | |
| PIN2022.11 | 48.4 | — | — | — | — | — | — | 97.02 | — | — | — | — | — | — | — | — | — | |
| TOD-BERTApproach category=Classification-based2026.03 | 48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRADE2023.05 | 46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRADE2023.05 | 46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRADE2022.11 | 45.6 | — | — | — | — | — | — | 96.55 | — | — | — | — | — | — | — | — | — | |
| TRADEHistory=Full2021.08 | 45.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRADEPretrained-Model=N, Parameters=N/A2021.09 | 45.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ChatGPT (GPT-4)Approach=FNCTOD, Backbone=gpt-4-1106-preview, Shots=02024.02 | 38.71 | — | 45.15 | 63.18 | 76.39 | 69.48 | 62.59 | — | 58.77 | 81.84 | 85.07 | 91.06 | 87.73 | 90.16 | — | — | — | |
| ChatGPT (GPT-3.5)Approach=FNCTOD, Backbone=gpt-3.5-turbo-1106, Shots=02024.02 | 38.56 | — | 37.56 | 60.12 | 74.43 | 67.29 | 61.31 | — | 67.15 | 87.2 | 82.86 | 90.21 | 86.9 | 92.48 | — | — | — | |
| FNCTOD-LLAMA2-13BApproach=FNCTOD, Model Type=Fine-tuned, Shots=02024.02 | 37.67 | — | 46.83 | 60.27 | 67.48 | 60.9 | 59.54 | — | 62.24 | 84.99 | 85.39 | 88.69 | 80.39 | 89.88 | — | — | — | |
| BAICHUAN2-13B-CHATApproach=FNCTOD, Shots=52024.02 | 33.21 | — | 40.15 | 59.02 | 69.31 | 60.67 | 56.56 | — | 53.67 | 79.57 | 81.36 | 87.82 | 81.95 | 89.45 | — | — | — | |
| ZEPHYR-7B-BETAApproach=FNCTOD, Shots=52024.02 | 32.11 | — | 38.43 | 63.18 | 74.1 | 56.2 | 57.68 | — | 56.5 | 81.97 | 79.52 | 91.19 | 86.56 | 90 | — | — | — | |
| VICUNA-13B-v1.5Approach=FNCTOD, Shots=52024.02 | 31.84 | — | 38.43 | 56.44 | 69.11 | 58.82 | 55.41 | — | 54.25 | 80.99 | 79.96 | 87.26 | 83.37 | 89.26 | — | — | — | |
| Heck et al. (2023)Backbone=GPT-3.52024.02 | 31.5 | — | 42 | 55.8 | 70.9 | 60.8 | 56.44 | — | 52.7 | — | — | — | — | — | — | — | — |