# Learning What Helps: Task-Aligned Context Selection for Vision Tasks

Source: [Learning What Helps: Task-Aligned Context Selection for Vision Tasks](<https://www.sota2.com/research/paper/692e72a4d6b0dd3342932391>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-03T05:02:38.198Z

Authors: Jingyu Guo, Emir Konuk, Fredrik Strand, Christos Matsoukas, Kevin Smith

Published: 2025-11-29T13:47:35.000Z

## Abstract

Humans often resolve visual uncertainty by comparing an image with relevant examples, but ViTs lack the ability to identify which examples would improve their predictions. We present Task-Aligned Context Selection (TACS), a framework that learns to select paired examples which truly improve task performance rather than those that merely appear similar. TACS jointly trains a selector network with the task model through a hybrid optimization scheme combining gradient-based supervision and reinforcement learning, making retrieval part of the learning objective. By aligning selection with task rewards, TACS enables discriminative models to discover which contextual examples genuinely help. Across 18 datasets covering fine-grained recognition, medical image classification, and medical image segmentation, TACS consistently outperforms similarity-based retrieval, particularly in challenging or data-limited settings.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2512.00489>)

[PDF](<https://arxiv.org/pdf/2512.00489.pdf>)