Source-Free Domain Adaptation with Vision-Language Prior
About
Source-Free Domain Adaptation (SFDA) seeks to adapt a source model, which is pre-trained on a supervised source domain, for a target domain, with only access to unlabeled target training data. Relying on pseudo labeling and/or auxiliary supervision, conventional methods are inevitably error-prone. To mitigate this limitation, in this work we for the first time explore the potentials of off-the-shelf vision-language (ViL) multimodal models (e.g., CLIP) with rich whilst heterogeneous knowledge. We find that directly applying the ViL model to the target domain in a zero-shot fashion is unsatisfactory, as it is not specialized for this particular task but largely generic. To make it task-specific, we propose a novel DIFO++ approach. Specifically, DIFO++ alternates between two steps during adaptation: (i) Customizing the ViL model by maximizing the mutual information with the target model in a prompt learning manner, (ii) Distilling the knowledge of this customized ViL model to the target model, centering on gap region reduction. During progressive knowledge adaptation, we first identify and focus on the gap region, where enclosed features are entangled and class-ambiguous, as it often captures richer task-specific semantics. Reliable pseudo-labels are then generated by fusing predictions from the target and ViL models, supported by a memory mechanism. Finally, gap region reduction is guided by category attention and predictive consistency for semantic alignment, complemented by referenced entropy minimization to suppress uncertainty. Extensive experiments show that DIFO++ significantly outperforms the state-of-the-art alternatives. Our code and data are available at https://github.com/tntek/DIFO-Plus.
Key insights
What problem does DIFO++ solve?
Source-Free Domain Adaptation adapts a labeled-source model to unlabeled target data without source access. DIFO++ uses a frozen vision-language model (CLIP) as external knowledge, customizing it via prompt learning and distilling task-specific knowledge to the target model.
How does DIFO++ differ from prior work?
Unlike prior SFDA methods relying solely on source-model and target-data knowledge, DIFO++ adds a vision-language prior. It introduces gap-region-driven distillation using referenced entropy, unlike DIFO which treats all samples uniformly.
What practical use do these findings have?
DIFO++ tests image classifiers that must adapt without retaining their original training data. Its Office, VisDA and DomainNet experiments cover changing visual domains and label sets, providing evidence for adaptation research rather than field deployment.
What limitations do the authors note?
DIFO++ requires white-box access to the ViL model for prompt learning, precluding black-box deployment. It lacks an anti-forgetting mechanism for continual learning, and its reliance on historical information limits online test-time adaptation use.