# Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts

Source: [Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts](<https://www.sota2.com/research/paper/68fa73a81e209131cea5db98>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:54:13.197Z

Authors: Xiaobo Guo, Soroush Vosoughi

Published: 2024-02-16T10:35:18.000Z

## Abstract

Aspect-based summarization has seen significant advancements, especially in structured text. Yet, summarizing disordered, large-scale texts, like those found in social media and customer feedback, remains a significant challenge. Current research largely targets predefined aspects within structured texts, neglecting the complexities of dynamic and disordered environments. Addressing this gap, we introduce Disordered-DABS, a novel benchmark for dynamic aspect-based summarization tailored to unstructured text. Developed by adapting existing datasets for cost-efficiency and scalability, our comprehensive experiments and detailed human evaluations reveal that Disordered-DABS poses unique challenges to contemporary summarization models, including state-of-the-art language models such as GPT-3.5.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Dynamic Aspect-Based Summarization on D-WikiHow (test)](<https://www.sota2.com/research/sota/dynamic-aspect-based-summarization-on-d-wikihow-test.md>): Dynamic Aspect-Based Summarization; D-WikiHow (test); #AbsAspDiff: 5.5.

- [Dynamic Aspect-Based Summarization on D-CnnDM (test)](<https://www.sota2.com/research/sota/dynamic-aspect-based-summarization-on-d-cnndm-test.md>): Dynamic Aspect-Based Summarization; D-CnnDM (test); #AbsAspDiff: 1.3.

- [Dynamic Aspect-Based Summarization on OASUM (test)](<https://www.sota2.com/research/sota/dynamic-aspect-based-summarization-on-oasum-test.md>): Dynamic Aspect-Based Summarization; OASUM (test); AbsAspDiff: 1.

- [Aspect-based Summarization on D-CnnDM (test)](<https://www.sota2.com/research/sota/aspect-based-summarization-on-d-cnndm-test.md>): Aspect-based Summarization; D-CnnDM (test); Coherence: 3.5.

- [Aspect Number Prediction on D-CnnDM (test)](<https://www.sota2.com/research/sota/aspect-number-prediction-on-d-cnndm-test.md>): Aspect Number Prediction; D-CnnDM (test); Absolute Aspect Number Difference: 1.28.

- [Aspect Number Prediction on D-WikiHow (test)](<https://www.sota2.com/research/sota/aspect-number-prediction-on-d-wikihow-test.md>): Aspect Number Prediction; D-WikiHow (test); Abs Aspect Number Difference: 2.69.

- [Aspect Number Prediction on OASUM (test)](<https://www.sota2.com/research/sota/aspect-number-prediction-on-oasum-test.md>): Aspect Number Prediction; OASUM (test); Absolute Aspect Number Difference: 1.48.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2402.10554>)

[PDF](<https://arxiv.org/pdf/2402.10554.pdf>)