# Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents

Source: [Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents](<https://www.sota2.com/research/paper/6a30d90ad33dfb23e783d1fc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-01T05:00:04.992Z

Authors: Shiyang Chen

Published: 2026-06-15T07:58:56.000Z

## Abstract

LLM agents mis-call tools, and the natural guess is that the model failed to see the right tool in a crowded harness. We show the opposite through a lens concurrent work sets aside -- the model's attention to labeled tool-definition segments. On real BFCL failures, by per-candidate attention argmax the model attends most to the correct tool 80% of the time (vs. 21% chance), and the gold is the under-attended segment on only 10%: it looks at the right tool and still picks wrong. This directly refutes the intuitive "crowded-harness / lost-in-the-middle" explanation: the failure is at the decision readout, not the harness, and we pin it there three ways. (1) Input vs. readout: repairing the prompt (reordering or duplicating the gold tool) recovers \<=23% of failures, while readout-side interventions recover 59-91%. (2) Representation-invariance: two gold-pointed interventions in different representations -- an additive attention-logit bias and a residual-stream steering vector -- recover largely the same failures (per-task Jaccard 0.865 pooled, 0.79-0.91 per model), so the bottleneck is localized to the readout independent of which representation is poked. (3) A training-free, gold-free selector: per-segment attention closes most of the gold-free-vs-oracle gap on BFCL (+11.9 pts pooled function-name selection vs. +17.9-pt oracle headroom) and adds +14.9 pts on Seal-Tools; every model positive (exact McNemar p\<=8e-4 each). Scopes differ: the causal attention-bias dose-response is bidirectional and monotonic on 10 mask-honoring models (3-32B), the full 0.5-32B span carrying only the correlational diagnostic; the deployable selector is evaluated on 5 single-turn models and does not yet transfer to a multi-turn loop.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Function-name selection on BFCL multiple live (300 tasks model)](<https://www.sota2.com/research/sota/function-name-selection-on-bfcl-multiple-live-300-tasks-model.md>): Function-name selection; BFCL multiple live (300 tasks model); Accuracy: 97.7.

- [Function-name selection on Seal-Tools single-call K=5 (294 tasks/model)](<https://www.sota2.com/research/sota/function-name-selection-on-seal-tools-single-call-k-5-294-tasks-model.md>): Function-name selection; Seal-Tools single-call K=5 (294 tasks/model); Accuracy: 93.2.

- [AST evaluation on BFCL live\_multiple](<https://www.sota2.com/research/sota/ast-evaluation-on-bfcl-live_multiple.md>): AST evaluation; BFCL live\_multiple; Free Name Accuracy: 94.3.

- [AST evaluation on Seal-Tools](<https://www.sota2.com/research/sota/ast-evaluation-on-seal-tools.md>): AST evaluation; Seal-Tools; Free Name Score: 94.2.

- [Per-step Tool Selection Prediction on τ-bench airline multi-step](<https://www.sota2.com/research/sota/per-step-tool-selection-prediction-on-bench-airline-multi-step.md>): per-step tool selection prediction; τ-bench airline multi-step; Success Rate: 100.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.16364>)

[PDF](<https://arxiv.org/pdf/2606.16364.pdf>)