# AgentAtlas: Beyond Outcome Leaderboards for LLM Agents

Source: [AgentAtlas: Beyond Outcome Leaderboards for LLM Agents](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-28T05:00:13.160Z

Authors: Parsa Mazaheri, Kasra Mazaheri

Published: 2026-05-19T22:05:12.000Z

## Abstract

Large language model agents now act on codebases, browsers, operating systems, calendars, files, and tool ecosystems, but their evaluations often collapse behavior into final task success. AgentAtlas reframes agent evaluation as a diagnostic vocabulary and audit protocol for separating outcome success from control-decision quality and trajectory quality. The paper contributes: (i) a six-state control-decision taxonomy (Act / Ask / Refuse / Stop / Confirm / Recover); (ii) a trajectory-failure vocabulary with primary error source and downstream impact; (iii) a 0/1/2 benchmark-coverage audit over fifteen agent benchmarks; and (iv) an illustrative protocol study on a synthetic 1,342-item set evaluated with eight models under taxonomy-aware and taxonomy-blind prompt formats. The synthetic demonstration is not a public benchmark release and should not be read as a definitive model comparison. Instead, it illustrates two measurement risks: mapped label agreement can change substantially when the explicit label menu is removed, and axis choice can change apparent rankings. AgentAtlas is intended to help benchmark designers state what behavior they cover, and to help evaluators diagnose failures that outcome-only leaderboards hide.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Control Calibration on Control](<https://www.sota2.com/research/sota/control-calibration-on-control.md>): Control calibration; Control; Accuracy: not recorded.

- [Tool-use security and utility evaluation on Security](<https://www.sota2.com/research/sota/tool-use-security-and-utility-evaluation-on-security.md>): Tool-use security and utility evaluation; Security; Utility Rate: not recorded.

- [Trajectory step classification on Trajectory nine-category label set](<https://www.sota2.com/research/sota/trajectory-step-classification-on-trajectory-nine-category-label-set.md>): Trajectory step classification; Trajectory nine-category label set; Label Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.20530>)

[PDF](<https://arxiv.org/pdf/2605.20530.pdf>)