# LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images

Source: [LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images](<https://www.sota2.com/research/paper/68fa71021e209131cea4e829>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T03:59:34.313Z

Authors: Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, Judy Hoffman

Published: 2023-05-30T16:09:16.000Z

## Abstract

We propose an automated algorithm to stress-test a trained visual model by generating language-guided counterfactual test images (LANCE). Our method leverages recent progress in large language modeling and text-based image editing to augment an IID test set with a suite of diverse, realistic, and challenging test images without altering model weights. We benchmark the performance of a diverse set of pre-trained models on our generated data and observe significant and consistent performance drops. We further analyze model sensitivity across different types of edits, and demonstrate its applicability at surfacing previously unknown class-level model biases in ImageNet. Code is available at https://github.com/virajprabhu/lance.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.19164>)

[PDF](<https://arxiv.org/pdf/2305.19164.pdf>)

[Implementation](<https://github.com/virajprabhu/lance>)