Image Classification on CIFAR-100 (val)
94.11AccuracyDoRA
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| DoRAModel=ViT Large, Params (%)=0.552025.10 | 94.11 | — | — | — | — | — | — | — | — | |
| LoRAModel=ViT Large, Params (%)=0.532025.10 | 94.06 | — | — | — | — | — | — | — | — | |
| PiSSAModel=ViT Large, Params (%)=0.532025.10 | 93.99 | — | — | — | — | — | — | — | — | |
| StelLAModel=ViT Large, Params (%)=0.542025.10 | 93.92 | — | — | — | — | — | — | — | — | |
| DoRAModel=ViT Base, Params (%)=0.752025.10 | 92.78 | — | — | — | — | — | — | — | — | |
| StelLAModel=ViT Base, Params (%)=0.732025.10 | 92.72 | — | — | — | — | — | — | — | — | |
| LoRAModel=ViT Base, Params (%)=0.722025.10 | 92.6 | — | — | — | — | — | — | — | — | |
| PiSSAModel=ViT Base, Params (%)=0.722025.10 | 92.28 | — | — | — | — | — | — | — | — | |
| Full fine-tuningFine-tuning regime=Full2022.03 | 91.8 | — | — | — | — | — | — | — | — | |
| Baseline with LoRAPre-training=ViT-MAE (pre-trained on ImageNet), Total Params (M)=309.72, Trainable Params (M)=6.42, Training Time (min)=116.50±12.61, Memory (GB-VRAM)=57.75±0.59, FLOPs (G)=60.94, Throughput (img/s)=88.78±0.49, Inference Time (s)=1.49±0.262026.03 | 91.8 | — | — | — | — | — | — | — | — | |
| Learnable Memory TokensMemory cells=102022.03 | 91.1 | — | — | — | — | — | — | — | — | |
| Learnable Memory TokensMemory cells=52022.03 | 91 | — | — | — | — | — | — | — | — | |
| Learnable Memory TokensMemory cells=12022.03 | 90.9 | — | — | — | — | — | — | — | — | |
| Learnable Memory TokensMemory cells=202022.03 | 90.9 | — | — | — | — | — | — | — | — | |
| BaselinePre-training=ViT-MAE (pre-trained on ImageNet), Total Params (M)=303.40, Trainable Params (M)=303.40, Training Time (min)=49.31±5.37, Memory (GB-VRAM)=49.59±1.46, FLOPs (G)=59.70, Throughput (img/s)=119.11±1.19, Inference Time (s)=1.02±0.152026.03 | 88.8 | — | — | — | — | — | — | — | — | |
| Head + Class token fine-tuningFine-tuning regime=Head + Class2022.03 | 88.6 | — | — | — | — | — | — | — | — | |
| Head-only fine-tuningFine-tuning regime=Head Only2022.03 | 85.8 | — | — | — | — | — | — | — | — | |
| CutMixBackbone=CAFormer-S12, Epochs=200, Batch size=1002025.10 | 84.69 | — | — | — | — | — | — | — | — | |
| MergeMixBackbone=CAFormer-S12, Epochs=200, Batch size=1002025.10 | 84.3 | — | — | — | — | — | — | — | — | |
| TransMixBackbone=CAFormer-S12, Epochs=200, Batch size=1002025.10 | 83.7 | — | — | — | — | — | — | — | — | |
| DeiTBackbone=CAFormer-S12, Epochs=200, Batch size=1002025.10 | 83.6 | — | — | — | — | — | — | — | — | |
| Reference Masking RegularizationBackbone=ViTb16, Pre-trained=ImageNet-1K, Adaptation Protocol=Linear Probing2024.06 | 82.78 | — | — | — | — | — | — | — | — | |
| MixUpBackbone=CAFormer-S12, Epochs=200, Batch size=1002025.10 | 81.64 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.0, Selection=Best2023.02 | 80.98 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.0, Epoch selection=Best2023.02 | 80.98 | — | — | — | — | — | — | — | — | |
| Proposed-MNoise rate=0.0, Epoch selection=Best2023.02 | 80.61 | — | — | — | — | — | — | — | — | |
| Proposed-LNoise rate=0.0, Selection=Best2023.02 | 80.44 | — | — | — | — | — | — | — | — | |
| Proposed-LNoise rate=0.0, Epoch selection=Best2023.02 | 80.44 | — | — | — | — | — | — | — | — | |
| StandardBackbone=ViTb16, Pre-trained=ImageNet-1K, Adaptation Protocol=Linear Probing2024.06 | 80.1 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.0, Selection=Last2023.02 | 79.71 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.0, Epoch selection=Last2023.02 | 79.71 | — | — | — | — | — | — | — | — | |
| Proposed-LNoise rate=0.1, Selection=Best2023.02 | 79.45 | — | — | — | — | — | — | — | — | |
| Teacher BaselineTeacher=ResNet32x4, Student=ShuffleNet-V12023.10 | 79.42 | — | — | — | — | — | — | — | — | |
| Teacher BaselineTeacher=ResNet32x4, Student=MobileNet-V22023.10 | 79.42 | — | — | — | — | — | — | — | — | |
| TeacherTeacher architecture=resnet32x42024.11 | 79.42 | — | — | — | — | — | — | — | — | |
| Proposed-MNoise rate=0.0, Epoch selection=Last2023.02 | 79.39 | — | — | — | — | — | — | — | — | |
| Teacher BaselineTeacher=ResNet50, Student=MobileNet-V22023.10 | 79.34 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.1, Selection=Best2023.02 | 79.32 | — | — | — | — | — | — | — | — | |
| DE-CENoise rate=0.0, Selection=Best2023.02 | 79.13 | — | — | — | — | — | — | — | — | |
| DE-CENoise rate=0.0, Epoch selection=Best2023.02 | 79.13 | — | — | — | — | — | — | — | — | |
| Proposed-LNoise rate=0.0, Selection=Last2023.02 | 79.03 | — | — | — | — | — | — | — | — | |
| Proposed-LNoise rate=0.0, Epoch selection=Last2023.02 | 79.03 | — | — | — | — | — | — | — | — | |
| LumiNetTeacher=WRN-40-2, Student=ShuffleNet-V12023.10 | 78.95 | — | — | — | — | — | — | — | — | |
| LumiNetTeacher=ResNet32x4, Student=ShuffleNet-V12023.10 | 78.88 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.1, Selection=Last2023.02 | 78.81 | — | — | — | — | — | — | — | — | |
| DYRNoise rate=0.0, Selection=Best2023.02 | 78.64 | — | — | — | — | — | — | — | — | |
| DYRNoise rate=0.0, Epoch selection=Best2023.02 | 78.64 | — | — | — | — | — | — | — | — | |
| TeacherArchitecture=ResNet-342021.02 | 78.6 | — | — | — | — | — | — | — | — | |
| LOOKAHEADBackbone=ResNet-18, Optimizer=LOOKAHEAD2019.07 | 78.34 | — | — | — | — | — | — | — | — | |
| DHKD + ReviewKDTeacher architecture=resnet32x4, Student architecture=resnet8x42024.11 | 78.29 | — | — | — | — | — | — | — | — | |
| SGDBackbone=ResNet-18, Optimizer=SGD2019.07 | 78.24 | — | — | — | — | — | — | — | — | |
| FCFDTeacher=ResNet32x4, Student=MobileNet-V22023.10 | 78.2 | — | — | — | — | — | — | — | — | |
| FCFDTeacher=ResNet-32x4, Student=MobileNet-V22023.10 | 78.2 | — | — | — | — | — | — | — | — | |
| AFDTeacher architecture=WRN-40-2, Student architecture=WRN-40-2, Architectural style=Same style2021.02 | 78.13 | — | — | — | — | — | — | — | — | |
| FCFDTeacher=ResNet32x4, Student=ShuffleNet-V12023.10 | 78.12 | — | — | — | — | — | — | — | — | |
| FCFDTeacher=ResNet-32x4, Student=ShuffleNet-V12023.10 | 78.12 | — | — | — | — | — | — | — | — | |
| BDCKDTeacher=ResNet32x4, Student=MobileNet-V22023.10 | 78.05 | — | — | — | — | — | — | — | — | |
| BDCKDTeacher=ResNet-32x4, Student=MobileNet-V22023.10 | 78.05 | — | — | — | — | — | — | — | — | |
| TeacherTeacher Backbone=ResNet34, Student Backbone=ResNet182023.05 | 78.05 | — | — | — | — | — | — | — | — | |
| SimKDTeacher architecture=resnet32x4, Student architecture=resnet8x4, Distillation manner=features2024.11 | 78.04 | — | — | — | — | — | — | — | — | |
| DYRNoise rate=0.0, Selection=Last2023.02 | 78.02 | — | — | — | — | — | — | — | — | |
| DYRNoise rate=0.0, Epoch selection=Last2023.02 | 78.02 | — | — | — | — | — | — | — | — | |
| POLYAKBackbone=ResNet-18, Optimizer=POLYAK2019.07 | 77.99 | — | — | — | — | — | — | — | — | |
| DHKD + ReviewKDTeacher architecture=WRN-40-2, Student architecture=WRN-16-22024.11 | 77.97 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.3x, Epoch selection=Best2023.02 | 77.92 | — | — | — | — | — | — | — | — | |
| Proposed-MNoise rate=0.3x, Epoch selection=Best2023.02 | 77.89 | — | — | — | — | — | — | — | — | |
| KDTeacher Backbone=ResNet34, Student Backbone=ResNet182023.05 | 77.87 | — | — | — | — | — | — | — | — | |
| LumiNetTeacher=ResNet32x4, Student=MobileNet-V22023.10 | 77.85 | — | — | — | — | — | — | — | — | |
| FCFDTeacher=WRN-40-2, Student=ShuffleNet-V12023.10 | 77.81 | — | — | — | — | — | — | — | — | |
| FCFDTeacher=WRN-40-2, Student=ShuffleNet-V12023.10 | 77.81 | — | — | — | — | — | — | — | — | |
| CRDTeacher architecture=WRN-40-2, Student architecture=WRN-40-2, Architectural style=Same style2021.02 | 77.8 | — | — | — | — | — | — | — | — | |
| ReviewKDTeacher=ResNet32x4, Student=MobileNet-V22023.10 | 77.78 | — | — | — | — | — | — | — | — | |
| ReviewKDTeacher=ResNet-32x4, Student=MobileNet-V22023.10 | 77.78 | — | — | — | — | — | — | — | — | |
| ATTTeacher architecture=WRN-40-2, Student architecture=WRN-40-2, Architectural style=Same style2021.02 | 77.78 | — | — | — | — | — | — | — | — | |
| FitNetTeacher architecture=WRN-40-2, Student architecture=WRN-40-2, Architectural style=Same style2021.02 | 77.66 | — | — | — | — | — | — | — | — | |
| KDTeacher architecture=WRN-40-2, Student architecture=WRN-40-2, Architectural style=Same style2021.02 | 77.63 | — | — | — | — | — | — | — | — | |
| RKDTeacher architecture=WRN-40-2, Student architecture=WRN-40-2, Architectural style=Same style2021.02 | 77.62 | — | — | — | — | — | — | — | — | |
| OursTeacher=ResNet-32x4, Student=MobileNet-V22023.10 | 77.55 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.4x, Epoch selection=Best2023.02 | 77.53 | — | — | — | — | — | — | — | — | |
| Proposed-MNoise rate=0.4x, Epoch selection=Best2023.02 | 77.51 | — | — | — | — | — | — | — | — | |
| OursTeacher=ResNet-32x4, Student=ResNet-8x42023.10 | 77.5 | — | — | — | — | — | — | — | — | |
| AFDTeacher architecture=ResNet-34, Student architecture=WRN-28-2, Architectural style=Different style2021.02 | 77.47 | — | — | — | — | — | — | — | — | |
| ReviewKDTeacher=ResNet32x4, Student=ShuffleNet-V12023.10 | 77.45 | — | — | — | — | — | — | — | — | |
| ReviewKDTeacher=ResNet-32x4, Student=ShuffleNet-V12023.10 | 77.45 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.4x, Epoch selection=Last2023.02 | 77.32 | — | — | — | — | — | — | — | — | |
| BDCKDTeacher=ResNet-32x4, Student=ResNet-8x42023.10 | 77.25 | — | — | — | — | — | — | — | — | |
| Single-CENoise rate=0.0, Epoch selection=Best2023.02 | 77.23 | — | — | — | — | — | — | — | — | |
| BDCKDTeacher=ResNet32x4, Student=ShuffleNet-V12023.10 | 77.2 | — | — | — | — | — | — | — | — | |
| BDCKDTeacher=ResNet-32x4, Student=ShuffleNet-V12023.10 | 77.2 | — | — | — | — | — | — | — | — | |
| ATTTeacher architecture=ResNet-34, Student architecture=WRN-28-2, Architectural style=Different style2021.02 | 77.2 | — | — | — | — | — | — | — | — | |
| Proposed-MNoise rate=0.4x, Epoch selection=Last2023.02 | 77.19 | — | — | — | — | — | — | — | — | |
| Proposed-LNoise rate=0.1, Selection=Last2023.02 | 77.15 | — | — | — | — | — | — | — | — | |
| ReviewKDTeacher=WRN-40-2, Student=ShuffleNet-V12023.10 | 77.14 | — | — | — | — | — | — | — | — | |
| ReviewKDTeacher=WRN-40-2, Student=ShuffleNet-V12023.10 | 77.14 | — | — | — | — | — | — | — | — | |
| StudentTeacher Backbone=ResNet34, Student Backbone=ResNet182023.05 | 77.1 | — | — | — | — | — | — | — | — | |
| StudentTeacher Backbone=VGG11, Student Backbone=ResNet182023.05 | 77.1 | — | — | — | — | — | — | — | — | |
| DKDTeacher=ResNet32x4, Student=MobileNet-V22023.10 | 77.07 | — | — | — | — | — | — | — | — | |
| DKDTeacher=ResNet-32x4, Student=MobileNet-V22023.10 | 77.07 | — | — | — | — | — | — | — | — | |
| DE-CENoise rate=0.3x, Epoch selection=Best2023.02 | 77.07 | — | — | — | — | — | — | — | — | |
| Proposed-LMNoise rate=0.3x, Epoch selection=Last2023.02 | 77.03 | — | — | — | — | — | — | — | — |