# COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes

Source: [COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes](<https://www.sota2.com/research/paper/68fa764b1e209131cea6c352>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:50:30.323Z

Authors: Koen Kraaijveld, Yifan Jiang, Kaixin Ma, Filip Ilievski

Published: 2024-09-06T06:49:55.000Z

## Abstract

While visual question-answering (VQA) benchmarks have catalyzed the development of reasoning techniques, they have focused on vertical thinking. Effective problem-solving also necessitates lateral thinking, which remains understudied in AI and has not been used to test visual perception systems. To bridge this gap, we formulate visual lateral thinking as a multiple-choice question-answering task and describe a three-step taxonomy-driven methodology for instantiating task examples. Then, we develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles based on publicly available collections of compounds and common phrases. COLUMBUS comprises over 1,000 puzzles, each with four answer candidates. While the SotA vision-language models (VLMs) achieve decent performance, our evaluation demonstrates a substantial gap between humans and models. VLMs benefit from human-curated descriptions but struggle to self-generate such representations at the right level of abstraction.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2409.04053>)

[PDF](<https://arxiv.org/pdf/2409.04053.pdf>)