# Scenario classification on Customer-service conversations Source: [Scenario classification on Customer-service conversations]() Collection index: [llms.txt]() Data updated: 2026-06-16T09:24:33.863Z Evaluation of Scenario Classification performance on the Customer-service conversations dataset, reporting F1 score, accuracy, and accuracy at top 3 predictions. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Scenario classification | | Dataset (including any stated split) | Customer-service conversations | | Metric | Scenario F1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 14 of 14 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Scenario F1](): Higher is better; unit: PERCENT. - [Scenario Accuracy](): Higher is better; unit: PERCENT. - [Accuracy@3](): Higher is better; unit: PERCENT. ## Recorded results | Method | Scenario F1 | Result date | Report paper | | --- | --- | --- | --- | | ELECTRA | 71.39 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | RoBERTa | 70.33 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | DeBERTa | 69.83 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | RoBERTa | 66.15 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | DeBERTa | 65.39 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | ELECTRA | 64.89 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | GPT-5.4 | 45.77 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | GPT-5.4 | 42.52 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | Qwen3-32B | 42.14 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | GPT-OSS-20B | 34.59 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | Qwen3-32B | 33.15 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | Llama-3.1-8B | 30.74 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | GPT-OSS-20B | 29.44 | Not recorded | 6a30d8fad33dfb23e783d0c4 | | Llama-3.1-8B | 15.56 | Not recorded | 6a30d8fad33dfb23e783d0c4 | ### ELECTRA Scenario F1: 71.39. Result ID: 6a31163ffbc373592c2db790. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"EARS"}. This record does not assert independent reproduction. ### RoBERTa Scenario F1: 70.33. Result ID: 6a31163ffbc373592c2db78f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"EARS"}. This record does not assert independent reproduction. ### DeBERTa Scenario F1: 69.83. Result ID: 6a31163ffbc373592c2db791. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"EARS"}. This record does not assert independent reproduction. ### RoBERTa Scenario F1: 66.15. Result ID: 6a31163ffbc373592c2db78c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"Transformer"}. This record does not assert independent reproduction. ### DeBERTa Scenario F1: 65.39. Result ID: 6a31163ffbc373592c2db78e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"Transformer"}. This record does not assert independent reproduction. ### ELECTRA Scenario F1: 64.89. Result ID: 6a31163ffbc373592c2db78d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"Transformer"}. This record does not assert independent reproduction. ### GPT-5.4 Scenario F1: 45.77. Result ID: 6a31163ffbc373592c2db78b. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Few-shot"}. This record does not assert independent reproduction. ### GPT-5.4 Scenario F1: 42.52. Result ID: 6a31163ffbc373592c2db787. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Zero-shot"}. This record does not assert independent reproduction. ### Qwen3-32B Scenario F1: 42.14. Result ID: 6a31163ffbc373592c2db78a. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Few-shot"}. This record does not assert independent reproduction. ### GPT-OSS-20B Scenario F1: 34.59. Result ID: 6a31163ffbc373592c2db789. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Few-shot"}. This record does not assert independent reproduction. ### Qwen3-32B Scenario F1: 33.15. Result ID: 6a31163ffbc373592c2db786. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Zero-shot"}. This record does not assert independent reproduction. ### Llama-3.1-8B Scenario F1: 30.74. Result ID: 6a31163ffbc373592c2db788. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Few-shot"}. This record does not assert independent reproduction. ### GPT-OSS-20B Scenario F1: 29.44. Result ID: 6a31163ffbc373592c2db785. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Zero-shot"}. This record does not assert independent reproduction. ### Llama-3.1-8B Scenario F1: 15.56. Result ID: 6a31163ffbc373592c2db784. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM Zero-shot"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.