# Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions Source: [Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions]() Collection index: [llms.txt]() Data updated: 2026-06-10T05:01:35.776Z Authors: Nemika Tyagi, Olga Kellert, Holly Hendrix, Nelvin Licona-Guevara, Justin Mackie, Phanos Kareen, Megan Michelle Smith, Tatiana Gallego Hernande, Samhitha Harish, Chitta Baral Published: 2026-02-06T02:02:07.000Z ## Abstract Spoken bilingual conversations pose substantial challenges for syntactic parsing because they often include disfluencies and discourse-driven structures that complicate dependency parsing under standard Universal Dependencies (UD) assumptions and evaluation practices. To systematically study these challenges, in this work, we first introduce a linguistically grounded taxonomy of conversational bilingual phenomena, together with SpokeBench, an expert-annotated English-Spanish benchmark for structurally complex speech. To address the limitations of existing evaluation practices, we propose Flex-UD, an ambiguity-aware evaluation metric that distinguishes catastrophic structural failures from linguistically acceptable variations. Finally, we introduce DECAP, a decoupled agentic parsing framework that separates spoken-phenomena handling from core syntactic analysis, enabling robust and interpretable dependency parsing without retraining. Experiments across both proprietary and open-weight LLMs show that DECAP substantially improves performance on complex conversational phenomena and achieves over 60% improvements in UPOS-F1 Score over baselines, while Flex-UD evaluations reveal gains that otherwise remain partially hidden under standard attachment-based metrics. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Universal Dependency Parsing on SpokeBench Contr. (EN) v1 (test)](): Universal Dependency Parsing; SpokeBench Contr. (EN) v1 (test); ID Score: 72.5. - [Universal Dependency Parsing on SpokeBench Contr. (ES) v1 (test)](): Universal Dependency Parsing; SpokeBench Contr. (ES) v1 (test); ID Score: 80. - [Syntactic Parsing on SpokeBench 1.0 (test)](): Syntactic Parsing; SpokeBench 1.0 (test); LAS: 0.39. - [Universal Dependency Parsing on SpokeBench Repetition v1 (test)](): Universal Dependency Parsing; SpokeBench Repetition v1 (test); ID Score: 72. - [Universal Dependency Parsing on SpokeBench Repetition+ v1 (test)](): Universal Dependency Parsing; SpokeBench Repetition+ v1 (test); ID Score: 70. - [Universal Dependency Parsing on SpokeBench Ellipses v1 (test)](): Universal Dependency Parsing; SpokeBench Ellipses v1 (test); ID Score: 51.8. - [Universal Dependency Parsing on SpokeBench Ellipses+ v1 (test)](): Universal Dependency Parsing; SpokeBench Ellipses+ v1 (test); ID Score: 60. - [Universal Dependency Parsing on SpokeBench Discourse v1 (test)](): Universal Dependency Parsing; SpokeBench Discourse v1 (test); ID Accuracy: 63.5. - [Universal Dependency Parsing on SpokeBench Discourse+ v1 (test)](): Universal Dependency Parsing; SpokeBench Discourse+ v1 (test); ID Accuracy: 60.4. - [Universal Dependency Parsing on SpokeBench Complex v1 (test)](): Universal Dependency Parsing; SpokeBench Complex v1 (test); ID Accuracy: 55.8. - [Universal Dependency Parsing on SpokeBench v1 (test)](): Universal Dependency Parsing; SpokeBench v1 (test); ID: 47.2. - [Universal Dependency Parsing on SpokeBench Overall v1 (test)](): Universal Dependency Parsing; SpokeBench Overall v1 (test); ID Accuracy: 58.2. ## Original sources and implementation [Original paper]() [PDF]()