# Visual Program Distillation with Template-Based Augmentation

Source: [Visual Program Distillation with Template-Based Augmentation](<https://www.sota2.com/research/paper/68fa78c41e209131cea7a638>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-11-06T05:01:36.493Z

Authors: Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem

Published: 2024-12-11T17:32:21.000Z

## Abstract

Adapting visual programming or prompting large language models (LLMs) to generate executable code for visual tasks like visual question answering (VQA) for specialized tasks or domains remains challenging due to high annotation and inference costs. We propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. We achieve this through synthetic data augmentation based on decoupling programs into higher-level skills, called templates, and their corresponding arguments. Experimental results show that, with a relatively small amount of question/answer data, small language models can generate high-quality specialized visual programs with the added benefit of much faster inference

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2412.08564>)

[PDF](<https://arxiv.org/pdf/2412.08564.pdf>)
