# Automated Instruction Revision (AIR): A Structured Comparison of Task Adaptation Strategies for LLM

Source: [Automated Instruction Revision (AIR): A Structured Comparison of Task Adaptation Strategies for LLM](<https://www.sota2.com/research/paper/69dc78a17d0172140ec78beb>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-14T05:00:11.146Z

Authors: Solomiia Bilyk, Volodymyr Getmanskyi, Taras Firman

Published: 2026-04-10T15:28:05.000Z

## Abstract

This paper studies Automated Instruction Revision (AIR), a rule-induction-based method for adapting large language models (LLMs) to downstream tasks using limited task-specific examples. We position AIR within the broader landscape of adaptation strategies, including prompt optimization, retrieval-based methods, and fine-tuning. We then compare these approaches across a diverse benchmark suite designed to stress different task requirements, such as knowledge injection, structured extraction, label remapping, and logical reasoning. The paper argues that adaptation performance is strongly task-dependent: no single method dominates across all settings. Across five benchmarks, AIR was strongest or near-best on label-remapping classification, while KNN retrieval performed best on closed-book QA, and fine-tuning dominated structured extraction and event-order reasoning. AIR is most promising when task behavior can be captured by compact, interpretable instruction rules, while retrieval and fine-tuning remain stronger in tasks dominated by source-specific knowledge or dataset-specific annotation regularities.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Classification (8 classes) on Twitter customer-support requests](<https://www.sota2.com/research/sota/classification-8-classes-on-twitter-customer-support-requests.md>): Classification (8 classes); Twitter customer-support requests; Accuracy (8 classes): 95.31.

- [Event Logical Reasoning on BizFinBench v2](<https://www.sota2.com/research/sota/event-logical-reasoning-on-bizfinbench-v2.md>): Event Logical Reasoning; BizFinBench v2; Accuracy: 51.67.

- [PII detection on PUPA](<https://www.sota2.com/research/sota/pii-detection-on-pupa.md>): PII detection; PUPA; F1 Score: 59.32.

- [Closed-book Question Answering on Ever Young](<https://www.sota2.com/research/sota/closed-book-question-answering-on-ever-young.md>): Closed-book Question Answering; Ever Young; LLM Score: 42.08.

- [Information Extraction on Campaign-finance filings](<https://www.sota2.com/research/sota/information-extraction-on-campaign-finance-filings.md>): Information Extraction; Campaign-finance filings; Mean per field Accuracy: 35.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.09418>)

[PDF](<https://arxiv.org/pdf/2604.09418.pdf>)