# Detecting Backdoor Samples in Contrastive Language Image Pretraining Source: [Detecting Backdoor Samples in Contrastive Language Image Pretraining]() Collection index: [llms.txt]() Data updated: 2025-10-26T04:04:15.628Z Authors: Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, James Bailey Published: 2025-02-03T14:21:05.000Z ## Abstract Contrastive language-image pretraining (CLIP) has been found to be vulnerable to poisoning backdoor attacks where the adversary can achieve an almost perfect attack success rate on CLIP models by poisoning only 0.01\\% of the training dataset. This raises security concerns on the current practice of pretraining large-scale models on unscrutinized web data using CLIP. In this work, we analyze the representations of backdoor-poisoned samples learned by CLIP models and find that they exhibit unique characteristics in their local subspace, i.e., their local neighborhoods are far more sparse than that of clean samples. Based on this finding, we conduct a systematic study on detecting CLIP backdoor attacks and show that these attacks can be easily and efficiently detected by traditional density ratio-based local outlier detectors, whereas existing backdoor sample detection methods fail. Our experiments also reveal that an unintentional backdoor already exists in the original CC3M dataset and has been trained into a popular open-source model released by OpenCLIP. Based on our detector, one can clean up a million-scale web dataset (e.g., CC3M) efficiently within 15 minutes using 4 Nvidia A100 GPUs. The code is publicly available in our \\href{https://github.com/HanxunH/Detect-CLIP-Backdoor-Samples}{GitHub repository}. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Adversarial Defense on ImageNet-1K 1.0 (test)](): Adversarial Defense; ImageNet-1k 1.0 (test); Classification Accuracy: not recorded. - [Backdoor Defense (Multimodal) on CC3M 60K subset 1.0 (test)](): Backdoor Defense (Multimodal); CC3M 60K subset 1.0 (test); Clean Accuracy (CA): not recorded. - [Unimodal Backdoor Defense on ImageNet-1K 1.0 (test)](): Backdoor Defense (Unimodal); ImageNet-1k 1.0 (test); Clean Accuracy (CA): not recorded. ## Original sources and implementation [Original paper]() [PDF]()