# Autoregressive Structured Prediction with Language Models Source: [Autoregressive Structured Prediction with Language Models]() Collection index: [llms.txt]() Data updated: 2025-12-25T15:17:24.928Z Authors: Tianyu Liu, Yuchen Jiang, Nicholas Monath, Ryan Cotterell, Mrinmaya Sachan Published: 2022-10-26T13:27:26.000Z ## Abstract Recent years have seen a paradigm shift in NLP towards using pretrained language models ({PLM}) for a wide range of tasks. However, there are many difficult design decisions to represent structures (e.g. tagged text, coreference chains) in a way such that they can be captured by PLMs. Prior work on structured prediction with PLMs typically flattens the structured output into a sequence, which limits the quality of structural information being learned and leads to inferior performance compared to classic discriminative models. In this work, we describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs, allowing in-structure dependencies to be learned without any loss. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at, namely, named entity recognition, end-to-end relation extraction, and coreference resolution. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Coreference Resolution on English OntoNotes 5.0 (test)](): Coreference Resolution; English OntoNotes 5.0 (test); MUC Precision: 86.1. - [Coreference Resolution on OntoNotes](): Coreference Resolution; OntoNotes; MUC: 87.2. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()