# Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use

Source: [Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use](<https://www.sota2.com/research/paper/6a45f097dc71372d7795a3c6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-03T05:00:11.862Z

Authors: Song-Lin Lv, Weiming Wu, Rui Zhu, Zi-Jian Cheng, Lan-Zhe Guo

Published: 2026-07-01T15:40:25.000Z

## Abstract

While Large Language Model (LLM) agents demonstrate proficiency in static benchmarks, their deployment in real-world scenarios is hindered by the dynamic nature of user queries, tool sets, and interaction dynamics. To address this generalization gap, we formalize OpenAgent (Tool-Use Agent in Open-World), a problem setting characterized by distributional shifts across query, action, observation, and domain dimensions. To systematically diagnose its impact, we construct a controlled sandbox environment where we define fine-grained environmental shifts across a four-tier hierarchy, Perception, Interaction, Reasoning, and Internalization, and conduct a comprehensive series of experiments. Our analysis yields a series of key insights, demonstrating that agents trained via both Supervised Fine-Tuning(SFT) and Reinforcement Learning suffer from varying degrees of performance degradation when confronting open environmental shifts. Building on these insights, we propose Perturbation-Augmented Fine-Tuning, a disturbance-based intervention strategy for SFT that lays the foundation for enhancing agent robustness and utility in realistic environments. Our code will be released at: https://github. com/LAMDA-NeSy/OpenAgent.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Logic Inversion on OpenAgent Tier-3](<https://www.sota2.com/research/sota/logic-inversion-on-openagent-tier-3.md>): Logic Inversion; OpenAgent Tier-3; Accuracy Delta: 22.7.

- [Semantic Trap on OpenAgent Tier-1](<https://www.sota2.com/research/sota/semantic-trap-on-openagent-tier-1.md>): Semantic Trap; OpenAgent Tier-1; Accuracy Delta: 28.6.

- [Tool Redirection on OpenAgent Tier-2](<https://www.sota2.com/research/sota/tool-redirection-on-openagent-tier-2.md>): Tool Redirection; OpenAgent Tier-2; Accuracy Delta: 26.5.

- [Fatal Error on OpenAgent Tier-4](<https://www.sota2.com/research/sota/fatal-error-on-openagent-tier-4.md>): Fatal Error; OpenAgent Tier-4; RR (Fatal Error Rate): 1.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.01084>)

[PDF](<https://arxiv.org/pdf/2607.01084.pdf>)