SLURP
Benchmarks
TaskDatasetSOTA resultTrendResults
SLURP Retain Set 1.0 (DR)
90.18IF
40
SLURP (test)
80.61F1 Score
26
SLURP (test)
90.07Accuracy (IC)
26
SLURP Forget Set (DF) 1.0
95.27IF
20
SLURP
89.16Accuracy
16
SLURP (test)
23.77WER
11
SLURP 6 tasks
84.41Avg Accuracy
9
SLURP 3 tasks
84.41Average Accuracy
9
SLURP Oracle
91.57Intent Accuracy
8
SLURP Reference transcriptions (test)
86.3SLU F1 Score
6
SLURP Whisper Medium transcriptions (test)
66.5SLU F1 Score
6
SLURP Whisper Small transcriptions (test)
64.6SLU-F1
6
SLURP Whisper Base transcriptions WER 26.4% (test)
59.1SLU F1 Score
6
SLURP Whisper Tiny transcriptions (test)
56.4SLU F1 Score
6
SLURP (test)
78.76Scenario Accuracy
5
SLURP 0 dB
33.29Intent Accuracy
4
SLURP 2.5 dB
46.68Intent Accuracy
4
SLURP 5 dB
59.47Intent Accuracy
4
SLURP 10 dB
74.94Intent Accuracy
4
SLURP Clean
86.02Intent Accuracy
4
SLURP (test)
90.68Intent Accuracy
4