# Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12

Source: [Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12](<https://www.sota2.com/research/paper/68fa7b6a1e209131cea884b0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:39:30.468Z

Authors: John Mendon\\c{c}a, Lining Zhang, Rahul Mallidi, Alon Lavie, Isabel Trancoso, Luis Fernando D'Haro, Jo\\~ao Sedoc

Published: 2025-09-16T22:13:45.000Z

## Abstract

The rapid advancement of Large Language Models (LLMs) has intensified the need for robust dialogue system evaluation, yet comprehensive assessment remains challenging. Traditional metrics often prove insufficient, and safety considerations are frequently narrowly defined or culturally biased. The DSTC12 Track 1, "Dialog System Evaluation: Dimensionality, Language, Culture and Safety," is part of the ongoing effort to address these critical gaps. The track comprised two subtasks: (1) Dialogue-level, Multi-dimensional Automatic Evaluation Metrics, and (2) Multilingual and Multicultural Safety Detection. For Task 1, focused on 10 dialogue dimensions, a Llama-3-8B baseline achieved the highest average Spearman's correlation (0.1681), indicating substantial room for improvement. In Task 2, while participating teams significantly outperformed a Llama-Guard-3-1B baseline on the multilingual safety subset (top ROC-AUC 0.9648), the baseline proved superior on the cultural subset (0.5126 ROC-AUC), highlighting critical needs in culturally-aware safety. This paper describes the datasets and baselines provided to participants, as well as submission evaluation results for each of the two proposed subtasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Dialogue response evaluation on DSTC 12 (test)](<https://www.sota2.com/research/sota/dialogue-response-evaluation-on-dstc-12-test.md>): Dialogue response evaluation; DSTC 12 (test); Empathy: 0.06.

- [Theme Distribution on Insurance Out-of-domain](<https://www.sota2.com/research/sota/theme-distribution-on-insurance-out-of-domain.md>): Theme Distribution; Insurance Out-of-domain; Acc: 41.5.

- [Theme Distribution on Finance Out-of-domain](<https://www.sota2.com/research/sota/theme-distribution-on-finance-out-of-domain.md>): Theme Distribution; Finance Out-of-domain; Accuracy: 24.6.

- [Theme Distribution on Banking](<https://www.sota2.com/research/sota/theme-distribution-on-banking.md>): Theme Distribution; BANKING; Acc: 0.368.

- [Theme Label Quality on Banking](<https://www.sota2.com/research/sota/theme-label-quality-on-banking.md>): Theme Label Quality; BANKING; ROUGE-1: 11.1.

- [Theme Label Quality on Finance Out-of-domain](<https://www.sota2.com/research/sota/theme-label-quality-on-finance-out-of-domain.md>): Theme Label Quality; Finance Out-of-domain; ROUGE-1: 5.

- [Theme Label Quality on Insurance Out-of-domain](<https://www.sota2.com/research/sota/theme-label-quality-on-insurance-out-of-domain.md>): Theme Label Quality; Insurance Out-of-domain; ROUGE-1: 12.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2509.13569>)

[PDF](<https://arxiv.org/pdf/2509.13569.pdf>)