Video Action Recognition on Kinetics-400 (val)
94.3Top-1 AccFTP-UniFormerV2-L/14
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FTP-UniFormerV2-L/14Pretrain data=CLIP-400M+K710-0.66M, Input=64 x 336^2, Crops=2x3, Params=402M, FLOPS=78.7T2024.03 | 94.3 | 99.4 | |
| FTP-UniFormerV2-L/14Pretrain data=CLIP-400M+K710-0.66M, Input=32 x 224^2, Crops=2x3, Params=402M, FLOPS=18.4T2024.03 | 93.4 | 99.3 | |
| FTP-UniFormerV2-B/16Pretrain data=CLIP-400M+K710-0.66M, Input=8 x 224^2, Crops=4x3, Params=136M, FLOPS=2.2T2024.03 | 91.9 | 98.9 | |
| InternVideoPretrain data=CLIP-400M+Hybrid-12M, Input=64 x 224^2, Crops=16x4, Params=1300M, FLOPS=86.2T2024.03 | 91.1 | 98.9 | |
| TubeVit-HPretrain data=IN-1K, Input=32 x 224^2, Crops=4x3, Params=635M, FLOPS=17.6T2024.03 | 90.9 | 98.9 | |
| VideoMAE V2-gPretrain data=Hybrid-1.35M, Input=64 x 266^2, Crops=2x3, Params=1050M, FLOPS=160.3T2024.03 | 90 | 98.4 | |
| UniFormerV2-L/14Pretrain data=CLIP-400M+K710-0.66M, Input=64 x 336^2, Crops=2x3, Params=354M, FLOPS=75.3T2024.03 | 90 | 98.4 | |
| MTV-HPretrain data=IN-21K+WTS-60M, Input=32 x 280^2, Crops=4x3, Params=1330M, FLOPS=73.6T2024.03 | 89.9 | 98.3 | |
| ATMBackbone=ViT-L/14 (336↑), Pre-training=Merged-2B, Frames x Crops x Clips=32x3x4, GFLOPs=3784x122023.07 | 89.4 | 98.3 | |
| UniFormerV2-L/14Pretrain data=CLIP-400M+K710-0.66M, Input=32 x 224^2, Crops=2x3, Params=354M, FLOPS=16.0T2024.03 | 89.3 | 98.2 | |
| MTV-HPretrain data=IN-21K+WTS-60M, Input=32 x 224^2, Crops=4x3, Params=1120M, FLOPS=44.5T2024.03 | 89.1 | 98.2 | |
| BIKEBackbone=ViT-L/14 (336↑)†, Pre-training=WIT-400M, Frames x Crops x Clips=32x4x3, GFLOPs=3728x122023.07 | 88.6 | 98.3 | |
| Text4VisBackbone=ViT-L/14 (336↑)†, Pre-training=WIT-400M, Frames x Crops x Clips=32x4x3, GFLOPs=3829x122023.07 | 88.4 | 98 | |
| ATMBackbone=ViT-L/14 (336↑), Pre-training=WIT-400M, Frames x Crops x Clips=32x3x4, GFLOPs=3784x122023.07 | 88.2 | 97.9 | |
| BIKE-L/14Evaluation Protocol=Full Fine-tuning, Pre-train=CLIP-400M, Frames x Cr. x Cl.=16x3x4, TFLOPS=0.8x122024.08 | 88.1 | 97.9 | |
| ATMBackbone=ViT-L/14, Pre-training=WIT-400M, Frames x Crops x Clips=32x3x4, GFLOPs=1684x122023.07 | 88 | 97.6 | |
| EVLBackbone=ViT-L/14 (336↑), Pre-training=WIT-400M, Frames x Crops x Clips=32x1x3, GFLOPs=181962023.07 | 87.7 | — | |
| X-CLIPBackbone=ViT-L/14 (336↑)†, Pre-training=WIT-400M, Frames x Crops x Clips=16x4x3, GFLOPs=3086x122023.07 | 87.7 | 97.4 | |
| EVL-L/14Pretrain data=CLIP-400M, Input=32 x 336^2, Crops=3x3, Params=67M, FLOPS=19.1T2024.03 | 87.7 | — | |
| X-CLIP-L/14Pretrain data=CLIP-400M, Input=16 x 336^2, Crops=4x3, Params=453M, FLOPS=37.0T2024.03 | 87.7 | — | |
| AIMBackbone=ViT-L/14, Pre-training=WIT-400M, Frames x Crops x Clips=32x1x3, GFLOPs=112082023.07 | 87.5 | 97.7 | |
| EVLBackbone=ViT-L/14, Pre-training=WIT-400M, Frames x Crops x Clips=32x1x3, GFLOPs=80882023.07 | 87.3 | — | |
| ATMBackbone=ViT-L/14, Pre-training=WIT-400M, Frames x Crops x Clips=8x3x4, GFLOPs=421x122023.07 | 87.3 | 97.4 | |
| ST-AdapterBackbone=ViT-L/14, Pre-training=WIT-400M, Frames x Crops x Clips=32x1x3, GFLOPs=82482023.07 | 87.2 | 97.6 | |
| TDS-CLIP-L/14Evaluation Protocol=Ours (Frozen CLIP), Pre-train=CLIP-400M, Frames x Cr. x Cl.=16x3x4, TFLOPS=1.6x122024.08 | 87.2 | 97.6 | |
| MViT-L↑352, 40×3 (MaskFeat)pre-train=MaskFeat, K600, FLOPs × views=3790 × 3 × 4, Param=2182021.12 | 87 | 97.4 | |
| MaskFeat-LPretrain data=IN-21K, Input=64 x 224^2, Crops=4x3, Params=218M, FLOPS=45.5T2024.03 | 87 | 97.4 | |
| SwinV2-G↑384pre-train=MIM + Sup. IN-21K+Ext-70M, FLOPs × views=N/A × 5 × 4, Param=30002021.12 | 86.8 | — | |
| MAE-HPretrain data=IN1K+K400+K600-1.9M, Input=16 x 224^2, Crops=4x3, Params=632M, FLOPS=25.1T2024.03 | 86.8 | 97.2 | |
| MViT-L↑352, 40×3 (MaskFeat)pre-train=MaskFeat, K400, FLOPs × views=3790 × 3 × 4, Param=2182021.12 | 86.7 | 97.3 | |
| TDS-CLIP-L/14Evaluation Protocol=Ours (Frozen CLIP), Pre-train=CLIP-400M, Frames x Cr. x Cl.=8x3x4, TFLOPS=0.8x122024.08 | 86.7 | 97.5 | |
| VideoMAE-HPretrain data=IN-21K, Input=16 x 224^2, Crops=5x3, Params=633M, FLOPS=17.9T2024.03 | 86.6 | 97.1 | |
| Florence 384pre-train=Text, FLD-900M, FLOPs × views=N/A × 3 × 4, Param=6472021.12 | 86.5 | 97.3 | |
| FlorenceBackbone=Florence (384↑), Pre-training=FLD-900M, Frames x Crops x Clips=32x4x3, GFLOPs=N/A2023.07 | 86.5 | 97.3 | |
| MViT-L↑312, 40×3 (MaskFeat)pre-train=MaskFeat, K400, FLOPs × views=2828 × 3 × 4, Param=2182021.12 | 86.4 | 97.1 | |
| MViT-L↑312, 32×3 (MaskFeat)pre-train=MaskFeat, K400, FLOPs × views=2063 × 3 × 5, Param=2182021.12 | 86.3 | 97.1 | |
| MViTv2-LPretrain data=IN-21K, Input=40 x 312^2, Crops=5x3, Params=218M, FLOPS=42.2T2024.03 | 86.1 | 97 | |
| MTVBackbone=MTV-H, Pre-training=JFT-300M, Frames x Crops x Clips=32x4x3, GFLOPs=3706x122023.07 | 85.8 | 96.6 | |
| UniFormerV2-B/16Pretrain data=CLIP-400M+K710-0.66M, Input=8 x 224^2, Crops=4x3, Params=115M, FLOPS=1.8T2024.03 | 85.6 | 97 | |
| Uniformer V2-B/16Evaluation Protocol=Full Fine-tuning, Pre-train=CLIP-400M, Frames x Cr. x Cl.=8x3x4, TFLOPS=0.2x122024.08 | 85.6 | 97 | |
| TokenLearnerpre-train=Sup., JFT-300M, FLOPs × views=4076 × 3 × 4, Param=4502021.12 | 85.4 | — | |
| TokenLearner-L/10Pretrain data=JFT-300M, Input=64 x 224^2, Crops=4x3, Params=450M, FLOPS=48.9T2024.03 | 85.4 | 96.3 | |
| DUALPATH-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x1, TFLOPS=0.7x32024.08 | 85.4 | 97.1 | |
| MViT-L↑312, 32×3pre-train=Sup., IN-21K, FLOPs × views=2063 × 3 × 5, Param=2182021.12 | 85.3 | 96.6 | |
| MViT-L, 16×4 (MaskFeat)pre-train=MaskFeat, K600, FLOPs × views=377 × 1 × 10, Param=2182021.12 | 85.1 | 96.6 | |
| Video Swin-LEvaluation Protocol=Full Fine-tuning, Pre-train=IN-21K, Frames x Cr. x Cl.=32x3x4, TFLOPS=0.6x122024.08 | 85.1 | 95.9 | |
| DiST-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x1, TFLOPS=0.7x32024.08 | 85 | 97 | |
| Swin-L↑384, 32×2pre-train=Sup., IN-21K, FLOPs × views=2107 × 5 × 10, Param=2002021.12 | 84.9 | 96.7 | |
| ViViT-Hpre-train=Sup., JFT-300M, FLOPs × views=3981 × 3 × 4, Param=6542021.12 | 84.9 | 95.8 | |
| Video Swin-LPretrain data=IN-21K, Input=32 x 384^2, Crops=10x5, Params=200M, FLOPS=105.4T2024.03 | 84.9 | 96.7 | |
| ViViT-HPretrain data=JFT-300M, Input=32 x 224^2, Crops=4x3, Params=654M, FLOPS=47.8T2024.03 | 84.9 | 95.8 | |
| ViViTBackbone=H/16x2, Pre-training=JFT-300M, Frames x Crops x Clips=32x4x3, GFLOPs=8316x122023.07 | 84.8 | 95.8 | |
| X-CLIP-B/16Evaluation Protocol=Full Fine-tuning, Pre-train=CLIP-400M, Frames x Cr. x Cl.=16x3x4, TFLOPS=0.6x122024.08 | 84.7 | 96.8 | |
| S-VIT-B/16Evaluation Protocol=Full Fine-tuning, Pre-train=CLIP-400M, Frames x Cr. x Cl.=16x3x4, TFLOPS=0.7x122024.08 | 84.7 | 96.8 | |
| AIM-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x1, TFLOPS=0.6x32024.08 | 84.7 | 96.7 | |
| TDS-CLIP-B/16Evaluation Protocol=Ours (Frozen CLIP), Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x4, TFLOPS=0.7x122024.08 | 84.5 | 96.7 | |
| MViT-L, 16×4 (MaskFeat)pre-train=MaskFeat, K400, FLOPs × views=377 × 1 × 10, Param=2182021.12 | 84.3 | 96.3 | |
| STAN-B/16Evaluation Protocol=Full Fine-tuning, Pre-train=CLIP-400M, Frames x Cr. x Cl.=16x3x1, TFLOPS=0.7x12024.08 | 84.2 | 96.5 | |
| Side4Video-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x4, TFLOPS=0.7x122024.08 | 84.2 | 96.5 | |
| EVL-B/16Views=32×3×1, GFLOPs=28132026.04 | 84.2 | — | |
| M2-CLIP-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x4, TFLOPS=0.8x122024.08 | 84.1 | 96.8 | |
| M2-CLIP-B/16Views=32×4×3, GFLOPs=101042026.04 | 84.1 | — | |
| PKS4-L/14Views=32×1×3, GFLOPs=7713.62026.04 | 84.1 | — | |
| AIM-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=8x3x1, TFLOPS=0.2x32024.08 | 83.9 | 96.3 | |
| TDS-CLIP-B/16Evaluation Protocol=Ours (Frozen CLIP), Pre-train=CLIP-400M, Frames x Cr. x Cl.=8x3x4, TFLOPS=0.2x122024.08 | 83.9 | 96.1 | |
| VideoMamba-M800eViews=32×4×3, GFLOPs=48362026.04 | 83.9 | — | |
| ActionCLIP-B/16Evaluation Protocol=Full Fine-tuning, Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x10, TFLOPS=0.6x302024.08 | 83.8 | 96.2 | |
| DiST-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=8x3x1, TFLOPS=0.2x32024.08 | 83.6 | 96.3 | |
| ViViT-Lpre-train=Sup., JFT-300M, FLOPs × views=3980 × 3 × 1, Param=3082021.12 | 83.5 | 94.3 | |
| MViT-L, 16×4pre-train=Sup., IN-21K, FLOPs × views=377 × 1 × 10, Param=2182021.12 | 83.5 | 95.9 | |
| Swin-L, 32×2pre-train=Sup., IN-21K, FLOPs × views=604 × 3 × 4, Param=1972021.12 | 83.1 | 95.9 | |
| SwinTransformer-L/14Views=32×4×3, GFLOPs=72482026.04 | 83.1 | — | |
| UniFormer V1-BPretrain data=IN-1K, Input=32 x 224^2, Crops=4x3, Params=50M, FLOPS=3.1T2024.03 | 83 | 95.4 | |
| MVIT-BEvaluation Protocol=Full Fine-tuning, Pre-train=Kinetics-600, Frames x Cr. x Cl.=32x5x1, TFLOPS=0.2x52024.08 | 82.9 | 95.7 | |
| EVL-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=8x3x1, TFLOPS=0.4x32024.08 | 82.9 | — | |
| Swin-B, 32×2pre-train=Sup., IN-21K, FLOPs × views=282 × 3 × 4, Param=882021.12 | 82.7 | 95.5 | |
| ST-Adapter-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=32x3x1, TFLOPS=0.6x32024.08 | 82.7 | 96.2 | |
| SwinTransformer-B/16Views=32×4×3, GFLOPs=33842026.04 | 82.7 | — | |
| ST-Adapter-B/16Views=32×3×1, GFLOPs=18212026.04 | 82.7 | — | |
| MViT-S, 16×4pre-train=Sup., IN-21K, FLOPs × views=71 × 1 × 10, Param=362021.12 | 82.6 | 95.3 | |
| VideoMamba-MViews=32×4×3, GFLOPs=48362026.04 | 82.4 | — | |
| MViT-S, 16×4 (MaskFeat)pre-train=MaskFeat, K400, FLOPs × views=71 × 1 × 10, Param=362021.12 | 82.2 | 95.1 | |
| MViT-L↑312, 32×3FLOPs × views=2063 × 3 × 5, Param=2182021.12 | 82.2 | 94.7 | |
| PKS4-B/16Views=32×1×3, GFLOPs=17452026.04 | 82.2 | — | |
| ST-Adapter-B/16Evaluation Protocol=Frozen CLIP, Pre-train=CLIP-400M, Frames x Cr. x Cl.=8x3x1, TFLOPS=0.2x32024.08 | 82 | 95.7 | |
| MTV-B/16Views=32×4×3, GFLOPs=4790.42026.04 | 81.8 | — | |
| ViViT FE-LEvaluation Protocol=Full Fine-tuning, Pre-train=IN-21K, Frames x Cr. x Cl.=128x3x1, TFLOPS=4.0x32024.08 | 81.7 | 93.8 | |
| ViViT-L/16×2 FEViews=32×1×3, GFLOPs=119402026.04 | 81.7 | — | |
| MoViNet-A6FLOPs × views=386 × 1 × 1, Param=312021.12 | 81.5 | 95.3 | |
| VideoMamba-SViews=32×4×3, GFLOPs=16202026.04 | 81.5 | — | |
| MViTv1-B, 64×3FLOPs × views=455 × 3 × 3, Param=372021.12 | 81.2 | 95.1 | |
| MViT-B, 64×3FLOPs (B) x Views=455 x 3 x 3, Params (M)=36.6, Training=from scratch2021.09 | 81.2 | 95.1 | |
| MViT-S, 16×4FLOPs × views=71 × 1 × 10, Param=362021.12 | 81.1 | 94.9 | |
| VATTPretrain=AS+HT, Modality=VAT, Protocol=Fine-tuned2022.11 | 81.1 | — | |
| Mformer-HRPretrain data=IN-21K, Input=16 x 336^2, Crops=10x3, Params=109M, FLOPS=28.8T2024.03 | 81.1 | 95.2 | |
| Mformer-L/14Views=16×3×10, GFLOPs=287642026.04 | 81.1 | — | |
| R3D-RS-200 + RAFLOPs (B) x Views=307 x 10 x 3, Params (M)=121.6, RandAugment=true, Input frames=48, Training=from scratch2021.09 | 81 | — | |
| ViT-B-TimeSformerpre-train=Sup., IN-21K, FLOPs × views=2380 × 3 × 1, Param=1212021.12 | 80.7 | 94.7 | |
| TimeSformer-LPretrain data=IN-21K, Input=96 x 224^2, Crops=10x3, Params=121M, FLOPS=7.1T2024.03 | 80.7 | 94.7 | |
| TimeSformer-LViews=96×1×3, GFLOPs=71402026.04 | 80.7 | — |