# SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering

Source: [SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering](<https://www.sota2.com/research/paper/68fa682b1e209131cea1a7b9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:11:21.285Z

Authors: Yung-Sung Chuang, Chi-Liang Liu, Hung-Yi Lee, Lin-shan Lee

Published: 2019-10-25T07:46:39.000Z

## Abstract

While various end-to-end models for spoken language understanding tasks have been explored recently, this paper is probably the first known attempt to challenge the very difficult task of end-to-end spoken question answering (SQA). Learning from the very successful BERT model for various text processing tasks, here we proposed an audio-and-text jointly learned SpeechBERT model. This model outperformed the conventional approach of cascading ASR with the following text question answering (TQA) model on datasets including ASR errors in answer spans, because the end-to-end model was shown to be able to extract information out of audio data before ASR produced errors. When ensembling the proposed end-to-end model with the cascade architecture, even better performance was achieved. In addition to the potential of end-to-end SQA, the SpeechBERT can also be considered for many other spoken language understanding tasks just as BERT for many text processing tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Spoken Question Answering on Spoken SQuAD (B) ASR (test)](<https://www.sota2.com/research/sota/spoken-question-answering-on-spoken-squad-b-asr-test.md>): Spoken Question Answering; Spoken SQuAD (B) ASR (test); EM: 60.37.

- [Spoken Question Answering on SQuAD lost (test)](<https://www.sota2.com/research/sota/spoken-question-answering-on-squad-lost-test.md>): Spoken Question Answering; SQuAD lost (test); F1 Score: 37.31.

- [Spoken Question Answering on Spoken SQuAD (test)](<https://www.sota2.com/research/sota/spoken-question-answering-on-spoken-squad-test.md>): Spoken Question Answering; Spoken SQuAD (test); Accuracy: not recorded.

- [Spoken Question Answering on Spoken SQuAD (A) Text (test)](<https://www.sota2.com/research/sota/spoken-question-answering-on-spoken-squad-a-text-test.md>): Spoken Question Answering; Spoken SQuAD (A) Text (test); EM: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1910.11559>)

[PDF](<https://arxiv.org/pdf/1910.11559.pdf>)