Active Speaker Detection on AVA-ActiveSpeaker (val)
98.8mAPLoCoNet + TalkNCE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| LoCoNet + TalkNCEbackbone=LoCoNet2023.09 | 98.8 | — | 99.3 | 3.2 | — | — | |
| TS-TalkNet2023.09 | 97.7 | — | 98.7 | 5.1 | — | — | |
| TalkNet2023.09 | 96.4 | — | 98.2 | 6 | — | — | |
| LoCoNet w/ CLcontrastive_learning=true, status=reproduced2025.01 | 95.5 | — | — | — | — | — | |
| TalkNCESingle=✗, Pre-training=✓, E2E=✓, Params(M)=34.3, FLOPs(G)=4.92026.07 | 95.5 | — | — | — | — | — | |
| LoCoNet w/ CL + LASERcontrastive_learning=true, LASER=true2025.01 | 95.4 | — | — | — | — | — | |
| LoCoNet + LASERLASER=true2025.01 | 95.3 | — | — | — | — | — | |
| LoCoNetType=End-to-End, Vid. Enc.=R-18+VTCN, Params=34.3M, average-FLOPs=0.51G2023.01 | 95.2 | — | 98 | — | — | — | |
| LoCoNetstatus=reproduced2025.01 | 95.2 | — | — | — | — | — | |
| LoCoNetSingle=✗, Pre-training=✓, E2E=✓, Params(M)=34.3, FLOPs(G)=4.92026.07 | 95.2 | — | — | — | — | — | |
| SPELL+Feature encoding network=2D ResNet-50-TSM [13, 20]2022.07 | 94.9 | — | — | — | — | — | |
| SPELL+Type=Multi-Stage, Vid. Enc.=R-50+TSM, Params=51.23M†, average-FLOPs=19.6G†2023.01 | 94.9 | — | — | — | — | — | |
| SPELL+Single=✗, Pre-training=✓, E2E=✗, Params(M)=47.3, FLOPs(G)=5.42026.07 | 94.9 | — | — | — | — | — | |
| LR-ASDSingle=✓, Pre-training=✗, E2E=✓, Params(M)=0.84, FLOPs(G)=0.512026.07 | 94.5 | — | — | — | — | — | |
| SPELLFeature encoding network=2D ResNet-18-TSM [13, 20]2022.07 | 94.2 | — | — | — | — | — | |
| SPELLSingle candidate?=false, Pre-training?=true, E2E?=false, Params(M)=22.5, FLOPs(G)=2.62023.03 | 94.2 | — | — | — | — | — | |
| SPELLType=Multi-Stage, Vid. Enc.=R-18+TSM, Params=23.5M†, average-FLOPs=8.7G†2023.01 | 94.2 | — | — | — | — | — | |
| SPELLSingle=✗, Pre-training=✓, E2E=✗, Params(M)=22.5, FLOPs(G)=2.62026.07 | 94.2 | — | — | — | — | — | |
| EASEE-50Backbone=ResNet50, Encoding=3D, Temporal Context=true, Supervision=A/V2022.03 | 94.1 | — | — | — | — | — | |
| EASEE-50Single candidate?=false, Pre-training?=true, E2E?=true, Params(M)=>74.7, FLOPs(G)=>65.52023.03 | 94.1 | — | — | — | — | — | |
| Light-ASDSingle candidate?=true, Pre-training?=false, E2E?=true, Params(M)=1.0, FLOPs(G)=0.6(0.2*3)2023.03 | 94.1 | — | — | — | — | — | |
| EASEEType=End-to-End, Vid. Enc.=3D R-50, Params=26.8M†, average-FLOPs=4.3G†2023.01 | 94.1 | — | — | — | — | — | |
| Light-ASDType=End-to-End, Vid. Enc.=(2+1)D Conv, Params=1.02M, average-FLOPs=0.2G2023.01 | 94.1 | — | — | — | — | — | |
| EASEE2025.01 | 94.1 | — | — | — | — | — | |
| EASEE-50Single=✗, Pre-training=✓, E2E=✓, Params(M)=>74.7, FLOPs(G)=>65.52026.07 | 94.1 | — | — | — | — | — | |
| TS-TalkNetType=End-to-End, Vid. Enc.=R-18+VTCN, Params=36.8M, average-FLOPs=2.3G2023.01 | 93.9 | — | — | — | — | — | |
| Light-ASD + LASERLASER=true2025.01 | 93.8 | — | — | — | — | — | |
| OursSingle=✓, Pre-training=✗, E2E=✓, Params(M)=1.02, FLOPs(G)=0.622026.07 | 93.8 | — | — | — | — | — | |
| Light-ASD*Single=✓, Pre-training=✗, E2E=✓, Params(M)=1.02, FLOPs(G)=0.622026.07 | 93.6 | — | — | — | — | — | |
| ASDNetBackbone=ResNext101, Encoding=3D, Temporal Context=true, Supervision=A/V2022.03 | 93.5 | — | — | — | — | — | |
| ASDNetFeature encoding network=3D ResNeXt-18 [17] + SincDSNet [27]2022.07 | 93.5 | — | — | — | — | — | |
| ASDNetSingle candidate?=false, Pre-training?=true, E2E?=false, Params(M)=51.3, FLOPs(G)=14.92023.03 | 93.5 | — | — | — | — | — | |
| ASDNetType=Multi-Stage, Vid. Enc.=3DRNext-101, Params=51.0M, average-FLOPs=13.2G†2023.01 | 93.5 | — | — | — | — | — | |
| Light-ASDstatus=reproduced2025.01 | 93.5 | — | — | — | — | — | |
| ASDNetTemporal Modeling=Bidirectional-GRU2021.06 | 93.5 | — | — | — | — | — | |
| ASDNetSingle=✗, Pre-training=✓, E2E=✗, Params(M)=51.3, FLOPs(G)=14.92026.07 | 93.5 | — | — | — | — | — | |
| EASEE-18Backbone=ResNet18, Encoding=3D, Temporal Context=true, Supervision=A/V2022.03 | 93.3 | — | — | — | — | — | |
| ADENetSingle candidate?=false, Pre-training?=false, E2E?=false, Params(M)=33.2, FLOPs(G)=22.8(7.6*3)2023.03 | 93.2 | — | — | — | — | — | |
| ADENetSingle=✓, Pre-training=✗, E2E=✓, Params(M)=33.2, FLOPs(G)=22.72026.07 | 93.2 | — | — | — | — | — | |
| ASD-TransformerSingle candidate?=false, Pre-training?=false, E2E?=false, Params(M)=>14.6, FLOPs(G)=>1.5(0.5*3)2023.03 | 93 | — | — | — | — | — | |
| ASD-TransType=End-to-End, Vid. Enc.=R-18+VTCN, Params=15.0M, average-FLOPs=0.55G2023.01 | 93 | — | — | — | — | — | |
| ASD-TransformerSingle=✓, Pre-training=✗, E2E=✓, Params(M)=>13.9, FLOPs(G)=>1.52026.07 | 93 | — | — | — | — | — | |
| Late-fusionComponents=GSCMIA and TalkNet2022.12 | 92.86 | — | — | — | — | — | |
| TalkNet + LASERLASER=true2025.01 | 92.5 | — | — | — | — | — | |
| TalkNetAudio=true, Visual=true2021.10 | 92.3 | — | — | — | — | — | |
| TalkNetBackbone=Custom, Encoding=3D+2D, Temporal Context=true, Supervision=A/V2022.03 | 92.3 | — | — | — | — | — | |
| TalkNetFeature encoding network=2D ResNet-18/34 [13] + 3D Conv2022.07 | 92.3 | — | — | — | — | — | |
| TalkNetSingle candidate?=true, Pre-training?=false, E2E?=false, Params(M)=15.7, FLOPs(G)=1.5(0.5*3)2023.03 | 92.3 | — | — | — | — | — | |
| TalkNetType=End-to-End, Vid. Enc.=R-18+VTCN, Params=15.7M, average-FLOPs=0.51G2023.01 | 92.3 | — | 96.8 | — | — | — | |
| TalkNetSingle=✓, Pre-training=✗, E2E=✓, Params(M)=15.7, FLOPs(G)=1.52026.07 | 92.3 | — | — | — | — | — | |
| UniConPre-training=true2021.08 | 92.2 | 0.97 | — | — | — | — | |
| UniConSingle candidate?=false, Pre-training?=true, E2E?=false, Params(M)=>22.4, FLOPs(G)=>1.82023.03 | 92.2 | — | — | — | — | — | |
| UniConType=End-to-End, Vid. Enc.=R-18, Params=23.8M, average-FLOPs=3.0G2023.01 | 92.2 | — | 97 | — | — | — | |
| TalkNetstatus=reproduced2025.01 | 92.2 | — | — | — | — | — | |
| UniConSingle=✗, Pre-training=✓, E2E=✗, Params(M)=>22.4, FLOPs(G)=>1.82026.07 | 92.2 | — | — | — | — | — | |
| UniConPre-training=false2021.08 | 92 | 0.97 | — | — | — | — | |
| UniConBackbone=Multiple, Encoding=2D, Temporal Context=true, Supervision=A/V2022.03 | 92 | — | — | — | — | — | |
| UniConFeature encoding network=2D ResNet-18 [13]2022.07 | 92 | — | — | — | — | — | |
| TalkNet2022.12 | 91.96 | — | — | — | — | — | |
| Our method2021.06 | 91.9 | — | 96.3 | — | — | — | |
| EASEE-2DBackbone=ResNet18, Encoding=2D, Temporal Context=true, Supervision=A/V2022.03 | 91.1 | — | — | — | — | — | |
| Causal ASDNetCausal=true2021.06 | 90.6 | — | — | — | — | — | |
| Sync-TalkNetSingle candidate?=true, Pre-training?=false, E2E?=false, Params(M)=>14.6, FLOPs(G)=>1.5(0.5*3)2023.03 | 89.8 | — | — | — | — | — | |
| Sync-TalkNetSingle=✓, Pre-training=✗, E2E=✓, Params(M)=15.7, FLOPs(G)=1.52026.07 | 89.8 | — | — | — | — | — | |
| EASEE-50Backbone=ResNet50, Encoding=3D, Temporal Context=false, Supervision=A/V, l=1, i=32022.03 | 89.6 | — | — | — | — | — | |
| VTPAudio=false, Visual=true2021.10 | 89.2 | — | — | — | — | — | |
| MAAS-TAN2021.01 | 88.8 | — | — | — | — | — | |
| MAAS-TANAudio=true, Visual=true2021.10 | 88.8 | — | — | — | — | — | |
| MAAS-TANBackbone=ResNet18, Encoding=2D, Temporal Context=true, Supervision=A/V2022.03 | 88.8 | — | — | — | — | — | |
| MAAS-TANFeature encoding network=2D ResNet-18 [13]2022.07 | 88.8 | — | — | — | — | — | |
| MAASSingle candidate?=false, Pre-training?=true, E2E?=false, Params(M)=22.5, FLOPs(G)=2.82023.03 | 88.8 | — | — | — | — | — | |
| MAASType=Multi-Stage, Vid. Enc.=R-18, Params=23.0M, average-FLOPs=1.6G2023.01 | 88.8 | — | — | — | — | — | |
| MAAS2025.01 | 88.8 | — | — | — | — | — | |
| MAAS-TAN2021.06 | 88.8 | — | — | — | — | — | |
| MAASSingle=✗, Pre-training=✓, E2E=✗, Params(M)=22.5, FLOPs(G)=2.82026.07 | 88.8 | — | — | — | — | — | |
| Chung et al.Audio=true, Visual=true2021.10 | 87.8 | — | — | — | — | — | |
| Chung et al.2021.06 | 87.8 | — | — | — | — | — | |
| Active Speakers Context2020.05 | 87.1 | — | — | — | — | — | |
| ASCPre-training=true2021.08 | 87.1 | — | — | — | — | — | |
| Alcazar et al.2021.01 | 87.1 | — | — | — | — | — | |
| Alcazar et al.Audio=true, Visual=true2021.10 | 87.1 | — | — | — | — | — | |
| ASCBackbone=ResNet18, Encoding=2D, Temporal Context=true, Supervision=A/V2022.03 | 87.1 | — | — | — | — | — | |
| ASCFeature encoding network=2D ResNet-18 [13]2022.07 | 87.1 | — | — | — | — | — | |
| ASCSingle candidate?=false, Pre-training?=true, E2E?=false, Params(M)=23.5, FLOPs(G)=1.82023.03 | 87.1 | — | — | — | — | — | |
| ASCType=Multi-Stage, Vid. Enc.=R-18, Params=23.0M, average-FLOPs=1.0G2023.01 | 87.1 | — | — | — | — | — | |
| ASC2021.06 | 87.1 | — | — | — | — | — | |
| Active Speakers Context2021.06 | 87.1 | — | — | — | — | — | |
| ASCSingle=✗, Pre-training=✓, E2E=✗, Params(M)=23.5, FLOPs(G)=1.82026.07 | 87.1 | — | — | — | — | — | |
| Google Baseline2021.06 | 86.3 | — | 92 | — | — | — | |
| Chung et al.Architecture=Temporal Convolutions2020.05 | 85.5 | — | — | — | — | — | |
| VGG-TempConvPre-training=true2021.08 | 85.5 | — | — | — | — | — | |
| Chung et al. (Temporal Convolutions)2021.01 | 85.5 | — | — | — | — | — | |
| Chung et al.Backbone=Custom, Encoding=3D+2D, Temporal Context=true, Supervision=A/V2022.03 | 85.5 | — | — | — | — | — | |
| Chung et al.Feature encoding network=VGG-M [5] + 3D Conv2022.07 | 85.5 | — | — | — | — | — | |
| Naver Corp. (Temporal Convolutions)Architecture=Temporal Convolutions2021.06 | 85.5 | — | — | — | — | — | |
| Chung et al.Architecture=LSTM2020.05 | 85.1 | — | — | — | — | — | |
| VGG-LSTMPre-training=true2021.08 | 85.1 | — | — | — | — | — | |
| Alcazar et al. (Temporal Context)2021.01 | 85.1 | — | — | — | — | — | |
| Chung et al. (LSTM)2021.01 | 85.1 | — | — | — | — | — | |
| MAAS-LAN2021.01 | 85.1 | — | — | — | — | — |