# Towards In-context Scene Understanding

Source: [Towards In-context Scene Understanding](<https://www.sota2.com/research/paper/68fa71111e209131cea4f09c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:32:46.190Z

Authors: Ivana Bala\\v{z}evi\\'c, David Steiner, Nikhil Parthasarathy, Relja Arandjelovi\\'c, Olivier J. H\\'enaff

Published: 2023-06-02T16:42:04.000Z

## Abstract

In-context learning$\\unicode{x2013}$the ability to configure a model's behavior with different prompts$\\unicode{x2013}$has revolutionized the field of natural language processing, alleviating the need for task-specific models and paving the way for generalist models capable of assisting with any query. Computer vision, in contrast, has largely stayed in the former regime: specialized decoders and finetuning protocols are generally required to perform dense tasks such as semantic segmentation and depth estimation. In this work we explore a simple mechanism for in-context learning of such scene understanding tasks: nearest neighbor retrieval from a prompt of annotated features. We propose a new pretraining protocol$\\unicode{x2013}$leveraging attention within and across images$\\unicode{x2013}$which yields representations particularly useful in this regime. The resulting Hummingbird model, suitably prompted, performs various scene understanding tasks without modification while approaching the performance of specialists that have been finetuned for each task. Moreover, Hummingbird can be configured to perform new tasks much more efficiently than finetuned models, raising the possibility of scene understanding in the interactive assistant regime.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Semantic Segmentation on PASCAL VOC 2012 (test)](<https://www.sota2.com/research/sota/semantic-segmentation-on-pascal-voc-2012-test.md>): Semantic segmentation; PASCAL VOC 2012 (test); mIoU: 82.1.

- [Semantic Segmentation on PASCAL VOC 2012 (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-pascal-voc-2012-val.md>): Semantic segmentation; PASCAL VOC 2012 (val); Mean IoU: 77.3.

- [Semantic Segmentation on ADE20K (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-val.md>): Semantic segmentation; ADE20K (val); mIoU: 48.2.

- [Monocular Depth Estimation on NYU v2 (test)](<https://www.sota2.com/research/sota/monocular-depth-estimation-on-nyu-v2-test.md>): Monocular Depth Estimation; NYU v2 (test); Abs Rel: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2306.01667>)

[PDF](<https://arxiv.org/pdf/2306.01667.pdf>)
