Automatic Speech Recognition on LRS3 (test)
0.79WER (%)Llama-AVSR
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Llama-AVSREncoder=Whisper, Trainable Parameters (M)=42, Labeled Hours=17562024.09 | 0.79 | — | — | — | — | — | |
| Llama-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=1,759, Learning Paradigm=LLM-Based Models, Leverage pseudo-labels from VoxCeleb2=true2025.08 | 0.79 | — | — | — | — | — | |
| OursSL Input Supported=Yes, Single Model=Yes, Training Data (h)=433, Learning Paradigm=LLM-Based Models, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 0.79 | — | — | — | — | — | |
| auto-avsrEncoder=Conformer, Trainable Parameters (M)=243, Labeled Hours=19022024.09 | 1 | — | — | — | — | — | |
| auto-avsrEncoder=Conformer, Trainable Parameters (M)=243, Labeled Hours=34482024.09 | 1 | — | — | — | — | — | |
| Auto-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=1,902, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 1 | — | — | — | — | — | |
| Auto-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=3,448, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 1 | — | — | — | — | — | |
| BRAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=4332024.09 | 1.1 | — | — | — | — | — | |
| Llama-AVSREncoder=Whisper, Trainable Parameters (M)=42, Labeled Hours=4332024.09 | 1.1 | — | — | — | — | — | |
| Llama-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=433, Learning Paradigm=LLM-Based Models, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 1.1 | — | — | — | — | — | |
| USRSL Input Supported=No, Single Model=Yes, Training Data (h)=1,759, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=true2025.08 | 1.2 | — | — | — | — | — | |
| BRAVEnSL Input Supported=No, Single Model=No, Training Data (h)=433, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 1.2 | — | — | — | — | — | |
| AV-HuBERT (A/MFCC→AV)Backbone=Transformer-Large, Criterion=S2S, LM=false, Labeled data (hrs)=433, Unlabeled data (hrs)=17592022.01 | 1.3 | — | — | — | — | — | |
| AV-HUBERTEncoder=Transformer, Trainable Parameters (M)=325, Labeled Hours=4332024.09 | 1.3 | — | — | — | — | — | |
| AV-HuBERTSL Input Supported=No, Single Model=No, Training Data (h)=433, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 1.3 | — | — | — | — | — | |
| RAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=4332024.09 | 1.4 | — | — | — | — | — | |
| Llama-AVSREncoder=AV-HUBERT A, Trainable Parameters (M)=40, Labeled Hours=4332024.09 | 1.4 | — | — | — | — | — | |
| RAVEnSL Input Supported=No, Single Model=No, Training Data (h)=1,759, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=true2025.08 | 1.4 | — | — | — | — | — | |
| RAVEnSL Input Supported=No, Single Model=No, Training Data (h)=433, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 1.4 | — | — | — | — | — | |
| Hsu et al. (2021a) (A/MFCC→A)Backbone=Transformer-Large, Criterion=S2S, LM=false, Labeled data (hrs)=433, Unlabeled data (hrs)=17592022.01 | 1.5 | — | — | — | — | — | |
| Llama-AVSREncoder=Whisper, Trainable Parameters (M)=42, Labeled Hours=302024.09 | 1.5 | — | — | — | — | — | |
| u-HuBERTSL Input Supported=No, Single Model=Yes, Training Data (h)=433, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 1.5 | — | — | — | — | — | |
| Fast ConformerEncoder=Conformer, Trainable Parameters (M)=197, Labeled Hours=4352024.09 | 1.6 | — | — | — | — | — | |
| BRAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=302024.09 | 1.7 | — | — | — | — | — | |
| RAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=302024.09 | 1.9 | — | — | — | — | — | |
| AV-HuBERT (A/MFCC→AV)Backbone=Transformer-Base, Criterion=S2S, LM=false, Labeled data (hrs)=433, Unlabeled data (hrs)=17592022.01 | 2 | — | — | — | — | — | |
| DistillAVModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 2.2 | — | — | — | — | — | |
| Ma et al. (2021b)Backbone=Conformer, Criterion=CTC+S2S, LM=true, Labeled data (hrs)=4332022.01 | 2.3 | — | — | — | — | — | |
| CM-seq2seqEncoder=Conformer, Trainable Parameters (M)=250, Labeled Hours=4332024.09 | 2.3 | — | — | — | — | — | |
| Whisper-finetunedEncoder=Whisper, Trainable Parameters (M)=1550, Labeled Hours=4332024.09 | 2.3 | — | — | — | — | — | |
| Hsu et al. (2021a) (A/MFCC→A)Backbone=Transformer-Base, Criterion=S2S, LM=false, Labeled data (hrs)=433, Unlabeled data (hrs)=17592022.01 | 2.4 | — | — | — | — | — | |
| DistillAVModel Complexity=Large, Unlabeled Data=433h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 2.4 | — | — | — | — | — | |
| DistillAVModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 2.5 | — | — | — | — | — | |
| RAVenModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=671M, Criterion=CTC+CE2025.02 | 2.6 | — | — | — | — | — | |
| AV-data2vecModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 2.7 | — | — | — | — | — | |
| AV-HuBERT (A/MFCC→AV)Backbone=Transformer-Large, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=17592022.01 | 2.9 | — | — | — | — | — | |
| AV-HuBERTModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 2.9 | — | — | — | — | — | |
| DistillAVModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 3 | — | — | — | — | — | |
| Hsu et al. (2021a) (A/MFCC→A)Backbone=Transformer-Large, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=17592022.01 | 3.2 | — | — | — | — | — | |
| AV-data2vecModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 3.7 | — | — | — | — | — | |
| AV-data2vecModel Complexity=Large, Unlabeled Data=433h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 3.7 | — | — | — | — | — | |
| AV-HuBERT (A/MFCC→AV)Backbone=Transformer-Base, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=17592022.01 | 3.8 | — | — | — | — | — | |
| RAVenModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=97M, Criterion=CTC+CE2025.02 | 3.8 | — | — | — | — | — | |
| AV-HuBERT (A/MFCC→AV)Backbone=Transformer-Large, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=4332022.01 | 4.2 | — | — | — | — | — | |
| AV-data2vecModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 4.4 | — | — | — | — | — | |
| Hsu et al. (2021a) (A/MFCC→A)Backbone=Transformer-Large, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=4332022.01 | 4.5 | — | — | — | — | — | |
| AV-HuBERTModel Complexity=Large, Unlabeled Data=433h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 4.5 | — | — | — | — | — | |
| AV-HuBERTModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 4.6 | — | — | — | — | — | |
| RAVenModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=97M, Criterion=CTC+CE2025.02 | 4.7 | — | — | — | — | — | |
| RNN-TSL Input Supported=No, Single Model=Yes, Training Data (h)=31,000, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 4.8 | — | — | — | — | — | |
| AV-HuBERT (A/MFCC→AV)Backbone=Transformer-Base, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=4332022.01 | 4.9 | — | — | — | — | — | |
| AV-HuBERTModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 4.9 | — | — | — | — | — | |
| Hsu et al. (2021a) (A/MFCC→A)Backbone=Transformer-Base, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=17592022.01 | 5 | — | — | — | — | — | |
| Hsu et al. (2021a) (A/MFCC→A)Backbone=Transformer-Base, Criterion=S2S, LM=false, Labeled data (hrs)=30, Unlabeled data (hrs)=4332022.01 | 5.4 | — | — | — | — | — | |
| AV2vec-MLMModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 5.6 | — | — | — | — | — | |
| Xu et al. (2020)Backbone=RNN, Criterion=S2S, LM=false, Labeled data (hrs)=4332022.01 | 7.2 | — | — | — | — | — | |
| Afouras et al. (2018a)Backbone=Transformer, Criterion=S2S, LM=true, Labeled data (hrs)=13622022.01 | 8.3 | — | — | — | — | — | |
| Afouras et al. (2018a)Backbone=Transformer, Criterion=CTC, LM=true, Labeled data (hrs)=13622022.01 | 8.9 | — | — | — | — | — | |
| Afouras et al.Training Modality=V, Labeled Data Dataset (hr)=LRS2, LRS3, MV-LRS, TEDx (2,676)2022.07 | — | — | — | 30.7 | — | — | |
| Afouras et al.Unlabeled A data (hr)=334, Training Modality=V, Labeled Data Dataset (hr)=LRW, LRS3 (590)2022.07 | — | — | — | 59.8 | — | — | |
| Auto-AVSRUnlabeled data=-, Labeled data=1.9kh, Shared across tasks=false2025.10 | — | — | — | — | 1 | — | |
| Auto-AVSRUnlabeled data=-, Labeled data=3.5kh, Shared across tasks=false2025.10 | — | — | — | — | 1 | — | |
| Auto-AVSRAudio Encoder=Conformer, Visual Encoder=-, Training Data (h)=19022026.03 | — | — | — | — | 1 | — | |
| Auto-AVSRAudio Encoder=Conformer, Visual Encoder=-, Training Data (h)=34482026.03 | — | — | — | — | 1 | — | |
| AUTO-AVSRType=A, Extra Data=Yes, Total Hours=8182023.03 | — | — | — | — | 1.5 | — | |
| AUTO-AVSRType=A, Total Hours=19022023.03 | — | — | — | — | 1 | — | |
| AUTO-AVSRType=A, Total Hours=34482023.03 | — | — | — | — | 1 | — | |
| AV-data2vec-BUnlabeled data=433h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 2 | — | |
| AV-data2vec-BUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.7 | — | |
| AV-data2vec-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.4 | — | |
| AV-HuBERTAudio Encoder=Transformer, Visual Encoder=-, Training Data (h)=4332026.03 | — | — | — | — | 1.6 | — | |
| AV-HUBERTType=A, Total Hours=17592023.03 | — | — | — | — | 1.3 | — | |
| AV-HuBERT-LUnlabeled data=433h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 2.7 | — | |
| AV-HuBERT-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.3 | — | |
| AV2vec-MLMUnlabeled data=433h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 2.7 | — | |
| AVUR-LLMAudio Encoder=Whisper, Visual Encoder=-, Training Data (h)=302026.03 | — | — | — | — | 1.3 | — | |
| AVUR-LLMAudio Encoder=Whisper, Visual Encoder=-, Training Data (h)=4332026.03 | — | — | — | — | 1 | — | |
| AVUR-LLMAudio Encoder=Whisper, Visual Encoder=-, Training Data (h)=17592026.03 | — | — | — | — | 0.7 | — | |
| BRAVEn-BUnlabeled data=433h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.9 | — | |
| BRAVEn-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.2 | — | |
| CM-seq2seqType=A, Extra Data=No, Total Hours=4382023.03 | — | — | — | — | 2.3 | — | |
| CM-seq2seqAudio Encoder=Conformer, Visual Encoder=-, Training Data (h)=4332026.03 | — | — | — | — | 2.3 | — | |
| DistillAV-BUnlabeled data=433h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.9 | 44.8 | |
| DistillAV-BUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.9 | 42.2 | |
| DistillAV-LUnlabeled data=433h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.8 | 41.8 | |
| DistillAV-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.4 | 39.2 | |
| Fast ConformerAudio Encoder=Conformer, Visual Encoder=-, Training Data (h)=4352026.03 | — | — | — | — | 1.6 | — | |
| Llama-AVSRUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | — | — | — | — | 1.1 | — | |
| Llama-AVSRUnlabeled data=1759h, Labeled data=1759h, Shared across tasks=false2025.10 | — | — | — | — | 0.81 | — | |
| Llama-AVSRAudio Encoder=Whisper, Visual Encoder=-, Training Data (h)=302026.03 | — | — | — | — | 1.5 | — | |
| Llama-AVSRAudio Encoder=Whisper, Visual Encoder=-, Training Data (h)=4332026.03 | — | — | — | — | 1.1 | — | |
| Llama-AVSRAudio Encoder=Whisper, Visual Encoder=-, Training Data (h)=17592026.03 | — | — | — | — | 0.79 | — | |
| Ma et al.Training Modality=AV, Labeled Data Dataset (hr)=LRW, LRS3 (590)2022.07 | — | 2.3 | — | — | — | — | |
| Ma et al.Training Modality=A, Labeled Data Dataset (hr)=LRW, LRS3 (590)2022.07 | — | — | 2.3 | — | — | — | |
| Ma et al.Training Modality=V, Labeled Data Dataset (hr)=LRW, LRS3 (590)2022.07 | — | — | — | 43.3 | — | — | |
| Ma et al.Training Modality=V, Labeled Data Dataset (hr)=LRW, LRS2, LRS3, AVSpeech (1,459)2022.07 | — | — | — | 31.5 | — | — | |
| Makino et al.Training Modality=AV, Labeled Data Dataset (hr)=YT31k (31,000)2022.07 | — | 4.5 | — | — | — | — | |
| Makino et al.Training Modality=A, Labeled Data Dataset (hr)=YT31k (31,000)2022.07 | — | — | 4.8 | — | — | — | |
| Makino et al.Training Modality=V, Labeled Data Dataset (hr)=YT31k (31,000)2022.07 | — | — | — | 33.6 | — | — | |
| Makino et al.Unlabeled data=-, Labeled data=31k, Shared across tasks=false2025.10 | — | — | — | — | 4.8 | — |