Video Panoptic Segmentation on VIPSeg (val)
58.5VPQCAVIS (Ours)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| CAVIS (Ours)Backbone=ViT-L, Temporal refiner=true2024.07 | 58.5 | 63.1 | 54.5 | 56.1 | |
| DVIS++Backbone=ViT-L, Temporal refiner=true2024.07 | 58 | 61.2 | 55.2 | 56 | |
| Axial-VSbackbone=ConvNeXt V2-L, setting=offline, segmenter=Video-kMaX2023.11 | 58 | 58.8 | 57.2 | — | |
| Axial-VS w/ Video-kMaXevaluation mode=offline, backbone=ConvNeXtV2-L2023.11 | 58 | 58.8 | 57.2 | — | |
| Axial-VSbackbone=ConvNeXt V2-L, setting=online/near-online, segmenter=Video-kMaX2023.11 | 57.7 | 58.3 | 57.1 | — | |
| Axial-VS w/ Video-kMaXevaluation mode=online/near-online, backbone=ConvNeXtV2-L2023.11 | 57.7 | 58.3 | 57.1 | — | |
| DVISbackbone=Swin-L, setting=offline2023.11 | 57.6 | 59.9 | 55.5 | — | |
| DVISevaluation mode=offline, backbone=Swin-L2023.11 | 57.6 | 59.9 | 55.5 | — | |
| DVIS-DAQBackbone=VIT-L2024.03 | 57.4 | 60.4 | 54.7 | 52 | |
| DVIS-DAQBackbone=ViT-L, Pre-training=DINOv2, GFLOPs=2315, FPS=42026.02 | 57.4 | — | — | 52 | |
| DVIS-DAQBackbone=ViT-Adapter-L [9], PT=DINOv2 [39], Encoder=fine-tuned, GFLOPs=2315, FPS=42026.03 | 57.4 | — | — | 52 | |
| Axial-VSbackbone=ConvNeXt-L, setting=offline, segmenter=Video-kMaX2023.11 | 57.1 | 59.3 | 54.8 | — | |
| Axial-VS w/ Video-kMaXevaluation mode=offline, backbone=ConvNeXt-L2023.11 | 57.1 | 59.3 | 54.8 | — | |
| CAVIS (Ours)Backbone=ViT-L, Temporal refiner=false2024.07 | 56.9 | 60.1 | 54.2 | 51 | |
| CAVISBackbone=ViT-L, Pre-training=DINOv2, GFLOPs=2612, FPS=102026.02 | 56.9 | — | — | 51 | |
| CAVISBackbone=ViT-Adapter-L [9], PT=DINOv2 [39], Encoder=fine-tuned, GFLOPs=2612, FPS=102026.03 | 56.9 | — | — | 51 | |
| CAVISBackbone=ViT-Adapter-L [9], PT=DINOv3 [41], Encoder=frozen, GFLOPs=2612, FPS=92026.03 | 56.8 | — | — | 51.2 | |
| CAVISBackbone=ViT-Adapter-L [9], PT=DINOv2 [39], Encoder=frozen, GFLOPs=2612, FPS=102026.03 | 56.4 | — | — | 49 | |
| Axial-VSbackbone=ConvNeXt-L, setting=online/near-online, segmenter=Video-kMaX2023.11 | 56.2 | 58.4 | 54 | — | |
| Axial-VS w/ Video-kMaXevaluation mode=online/near-online, backbone=ConvNeXt-L2023.11 | 56.2 | 58.4 | 54 | — | |
| DVIS++Backbone=ViT-L, Temporal refiner=false2024.07 | 56 | 58 | 54.3 | 49.8 | |
| DVIS++Backbone=VIT-L2024.03 | 56 | 58 | 54.3 | 49.8 | |
| DVIS++Backbone=ViT-L, Pre-training=DINOv2, GFLOPs=2290, FPS=132026.02 | 56 | — | — | 49.8 | |
| DVIS++Backbone=ViT-Adapter-L [9], PT=DINOv2 [39], Encoder=fine-tuned, GFLOPs=2290, FPS=132026.03 | 56 | — | — | 49.8 | |
| PMTBackbone=ViT-L [14], PT=DINOv3 [41], Encoder=frozen, GFLOPs=2037, FPS=582026.03 | 55.5 | — | — | 49.2 | |
| PMTBackbone=ViT-L [14], PT=DINOv2 [39], Encoder=frozen, GFLOPs=2037, FPS=602026.03 | 55.3 | — | — | 48.2 | |
| VidEoMTBackbone=ViT-L, Pre-training=DINOv2, GFLOPs=1897, FPS=752026.02 | 55.2 | — | — | 48.9 | |
| VidEoMTBackbone=ViT-L [14], PT=DINOv2 [39], Encoder=fine-tuned, GFLOPs=1897, FPS=752026.03 | 55.2 | — | — | 48.9 | |
| VidEoMTBackbone=ViT-L [14], PT=DINOv3 [41], Encoder=fine-tuned, GFLOPs=1897, FPS=712026.03 | 55.1 | — | — | 48.1 | |
| DVISbackbone=Swin-L, setting=online/near-online2023.11 | 54.7 | 54.8 | 54.6 | — | |
| DVISevaluation mode=online/near-online, backbone=Swin-L2023.11 | 54.7 | 54.8 | 54.6 | — | |
| DVISBackbone=Swin-L2024.03 | 54.7 | 54.8 | 54.6 | 47.7 | |
| DVISBackbone=Swin-L, Pre-training=IN21K, GFLOPs=879, FPS=202026.02 | 54.7 | — | — | 47.7 | |
| DVISBackbone=Swin-L [26], PT=ImageNet-21K [13], Encoder=fine-tuned, GFLOPs=879, FPS=202026.03 | 54.7 | — | — | 47.7 | |
| Video-kMaXbackbone=ConvNeXt-L, setting=online/near-online2023.11 | 51.9 | — | — | — | |
| TarViSShared Model=true, Backbone=Swin-L2023.01 | 48 | 58.2 | 39 | 52.9 | |
| TarVISbackbone=Swin-L, setting=online/near-online2023.11 | 48 | 58.2 | 39 | — | |
| TarVISevaluation mode=online/near-online, backbone=Swin-L2023.11 | 48 | 58.2 | 39 | — | |
| TarVISBackbone=Swin-L2024.03 | 48 | 58.2 | 39 | 52.9 | |
| Axial-VSbackbone=ResNet50, setting=offline, segmenter=Video-kMaX2023.11 | 46.7 | 46.7 | 46.6 | — | |
| Axial-VS w/ Video-kMaXevaluation mode=offline, backbone=ResNet502023.11 | 46.7 | 46.7 | 46.6 | — | |
| Axial-VSbackbone=ResNet50, setting=online/near-online, segmenter=Video-kMaX2023.11 | 46.1 | 45.6 | 46.6 | — | |
| Axial-VS w/ Video-kMaXevaluation mode=online/near-online, backbone=ResNet502023.11 | 46.1 | 45.6 | 46.6 | — | |
| CAVIS (Ours)Backbone=ResNet-50, Temporal refiner=true2024.07 | 45.3 | 47.5 | 43.4 | 45.3 | |
| DVIS++Backbone=ResNet-50, Temporal refiner=true2024.07 | 44.2 | 44.5 | 43.9 | 43.6 | |
| DVISBackbone=ResNet-50, Temporal refiner=true2024.07 | 43.2 | 43.6 | 42.8 | 42.8 | |
| DVISbackbone=ResNet50, setting=offline2023.11 | 43.2 | 43.6 | 42.8 | — | |
| DVISevaluation mode=offline, backbone=ResNet502023.11 | 43.2 | 43.6 | 42.8 | — | |
| CAVIS (Ours)Backbone=ResNet-50, Temporal refiner=false2024.07 | 42.4 | 43.1 | 41.8 | 39.7 | |
| DVIS-DAQBackbone=R502024.03 | 42.1 | 41.7 | 42.5 | 39.3 | |
| DVIS++Backbone=ResNet-50, Temporal refiner=false2024.07 | 41.9 | 41 | 42.7 | 38.5 | |
| DVIS++Backbone=R502024.03 | 41.9 | 41 | 42.7 | 38.5 | |
| DVISBackbone=ResNet-50, Temporal refiner=false2024.07 | 39.4 | 38.6 | 40.1 | 36.3 | |
| DVISBackbone=R502024.03 | 39.4 | 38.6 | 40.1 | 36.3 | |
| Tube-LinkBackbone=ResNet-50, Temporal refiner=false2024.07 | 39.2 | — | — | 39.5 | |
| Tube-Linkbackbone=ResNet50, setting=online/near-online2023.11 | 39.2 | — | — | — | |
| DVISbackbone=ResNet50, setting=online/near-online2023.11 | 39.2 | 39.3 | 39 | — | |
| DVISevaluation mode=online/near-online, backbone=ResNet502023.11 | 39.2 | 39.3 | 39 | — | |
| Video-kMaxBackbone=ResNet-50, Temporal refiner=false2024.07 | 38.2 | — | — | 39.9 | |
| Video-kMaXbackbone=ResNet50, setting=online/near-online2023.11 | 38.2 | — | — | — | |
| TarViSShared Model=true, Backbone=Swin-T2023.01 | 35.8 | 42.7 | 29.7 | 45.3 | |
| TarVISBackbone=ResNet-50, Temporal refiner=false2024.07 | 33.5 | 39.2 | 28.5 | 43.1 | |
| TarViSShared Model=true, Backbone=R-502023.01 | 33.5 | 39.2 | 28.5 | 43.1 | |
| TarVISbackbone=ResNet50, setting=online/near-online2023.11 | 33.5 | 39.2 | 28.5 | — | |
| TarVISevaluation mode=online/near-online, backbone=ResNet502023.11 | 33.5 | 39.2 | 28.5 | — | |
| TarVISBackbone=R502024.03 | 33.5 | 39.2 | 28.5 | 43.1 | |
| TubeFormer-DeepLabbackbone=B32022.05 | 31.2 | — | — | 41.5 | |
| TubeFormerbackbone=Axial-ResNet50-B3, setting=online/near-online2023.11 | 31.2 | — | — | — | |
| TubeFormer-DeepLabbackbone=B12022.05 | 29.2 | — | — | 39.8 | |
| Video K-NetBackbone=ResNet-50, Temporal refiner=false2024.07 | 26.1 | — | — | 31.5 | |
| Video K-Netbackbone=ResNet50, setting=online/near-online2023.11 | 26.1 | — | — | — | |
| Clip-PanoFCNBackbone=ResNet-50, Temporal refiner=false2024.07 | 22.9 | 25 | 20.8 | 31.5 | |
| Clip-PanoFCN2022.05 | 22.9 | — | — | 31.5 | |
| Clip-PanoFCNShared Model=false2023.01 | 22.9 | 25 | 20.8 | 31.5 | |
| Clip-PanoFCNbackbone=ResNet50, setting=online/near-online2023.11 | 22.9 | — | — | — | |
| VPSNet-SiamTrackBackbone=ResNet-50, Temporal refiner=false2024.07 | 17.2 | 17.3 | 17.3 | 21.1 | |
| VPSNet-SiamTrackShared Model=false2023.01 | 17.2 | 17.3 | 17.3 | 21.1 | |
| VPSNet-SiamTrackbackbone=ResNet50, setting=online/near-online2023.11 | 17.2 | — | — | — | |
| VPSNet-FuseTrackbackbone=ResNet50, setting=online/near-online2023.11 | 17 | — | — | — | |
| VIP-DeeplabBackbone=ResNet-50, Temporal refiner=false2024.07 | 16 | 12.3 | 18.2 | 22 | |
| VIP-DeeplabShared Model=false2023.01 | 16 | 12.3 | 18.2 | 22 | |
| ViP-DeepLabbackbone=ResNet50, setting=online/near-online2023.11 | 16 | — | — | — | |
| VPSNetShared Model=false2023.01 | 14 | 14 | 14.2 | 20.8 |