# Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understanding

Source: [Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understanding](<https://www.sota2.com/research/paper/68fa703c1e209131cea49cce>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:36:15.259Z

Authors: Umberto Cappellazzo, Muqiao Yang, Daniele Falavigna, Alessio Brutti

Published: 2023-05-23T10:24:07.000Z

## Abstract

The ability to learn new concepts sequentially is a major weakness for modern neural networks, which hinders their use in non-stationary environments. Their propensity to fit the current data distribution to the detriment of the past acquired knowledge leads to the catastrophic forgetting issue. In this work we tackle the problem of Spoken Language Understanding applied to a continual learning setting. We first define a class-incremental scenario for the SLURP dataset. Then, we propose three knowledge distillation (KD) approaches to mitigate forgetting for a sequence-to-sequence transformer model: the first KD method is applied to the encoder output (audio-KD), and the other two work on the decoder output, either directly on the token-level (tok-KD) or on the sequence-level (seq-KD) distributions. We show that the seq-KD substantially improves all the performance metrics, and its combination with the audio-KD further decreases the average WER and enhances the entity prediction metric.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Spoken Language Understanding on SLURP 3 tasks](<https://www.sota2.com/research/sota/spoken-language-understanding-on-slurp-3-tasks.md>): Spoken Language Understanding; SLURP 3 tasks; Average Accuracy: 74.28.

- [Spoken Language Understanding on FSC 3 tasks](<https://www.sota2.com/research/sota/spoken-language-understanding-on-fsc-3-tasks.md>): Spoken Language Understanding; FSC 3 tasks; Avg Accuracy: 84.79.

- [Spoken Language Understanding on SLURP 6 tasks](<https://www.sota2.com/research/sota/spoken-language-understanding-on-slurp-6-tasks.md>): Spoken Language Understanding; SLURP 6 tasks; Avg Accuracy: 69.91.

- [Spoken Language Understanding on FSC (test)](<https://www.sota2.com/research/sota/spoken-language-understanding-on-fsc-test.md>): Spoken Language Understanding; FSC (test); Intent Accuracy: 73.65.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.13899>)

[PDF](<https://arxiv.org/pdf/2305.13899.pdf>)