# Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraints

Source: [Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraints](<https://www.sota2.com/research/paper/68fa7a611e209131cea82bc9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:19:03.690Z

Authors: Sunil Kumar, Bowen Zhao, Leo Dirac, Paulina Varshavskaya

Published: 2025-06-10T20:11:44.000Z

## Abstract

Despite tremendous recent advances in large model reasoning ability, vision-language models (VLMs) still struggle with detailed visual reasoning, especially when compute resources are limited. To address this challenge, we draw inspiration from methods like Deepseek-r1 for VLMs and train smaller-scale models with Group Relative Policy Optimization (GRPO) to use external tools such as zoom. The greatest benefit is obtained with a combination of GRPO learning, a simple reward structure, a simplified tool-calling interface, allocating additional tokens to the result of the tool call, and a training data mix that over-represents visually difficult examples. Compared to similarly-sized baseline models, our method achieves better performance on some visual question-answering (VQA) tasks, thanks to the detailed visual information gathered from the external tool.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual Reasoning on V\*Bench](<https://www.sota2.com/research/sota/visual-reasoning-on-v-bench.md>): Visual Reasoning; V\*Bench; Accuracy: 80.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2506.14821>)

[PDF](<https://arxiv.org/pdf/2506.14821.pdf>)
