# Tool-use security and utility evaluation on Security

Source: [Tool-use security and utility evaluation on Security](<https://www.sota2.com/research/sota/tool-use-security-and-utility-evaluation-on-security>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-21T11:04:55.847Z

Evaluation focusing on the security posture and general utility of tool-using agents. Metrics cover refusal behavior (safe vs. over-refusal), attack success rates, and overall utility performance.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Tool-use security and utility evaluation |
| Dataset (including any stated split) | Security |
| Metric | Utility Rate |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 16 of 16 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Utility Rate](<https://www.sota2.com/research/sota/tool-use-security-and-utility-evaluation-on-security.md?metric=Utility%20Rate>): Higher is better; unit: PERCENT.

- [Over-refusal Rate](<https://www.sota2.com/research/sota/tool-use-security-and-utility-evaluation-on-security.md?metric=Over-refusal%20Rate>): Lower is better; unit: PERCENT.

- [Attack Following Rate](<https://www.sota2.com/research/sota/tool-use-security-and-utility-evaluation-on-security.md?metric=Attack%20Following%20Rate>): Lower is better; unit: PERCENT.

- [Safe Refusal Rate](<https://www.sota2.com/research/sota/tool-use-security-and-utility-evaluation-on-security.md?metric=Safe%20Refusal%20Rate>): Higher is better; unit: PERCENT.

- [Legacy F1 Score](<https://www.sota2.com/research/sota/tool-use-security-and-utility-evaluation-on-security.md?metric=Legacy%20F1%20Score>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Utility Rate | Result date | Report paper |
| --- | --- | --- | --- |
| gpt-5.4-mini | 98.4 | Not recorded | 6a0e919cf984739ed6738b8c |
| Gemma-4-26B-A4B-it | 95.3 | Not recorded | 6a0e919cf984739ed6738b8c |
| Claude Sonnet 4.6 | 78.5 | Not recorded | 6a0e919cf984739ed6738b8c |
| Gemma-4-26B-A4B-it | 72.9 | Not recorded | 6a0e919cf984739ed6738b8c |
| gpt-oss-20B | 71.4 | Not recorded | 6a0e919cf984739ed6738b8c |
| Gemini 3.1 Flash Lite | 69.8 | Not recorded | 6a0e919cf984739ed6738b8c |
| gpt-oss-20B | 60.3 | Not recorded | 6a0e919cf984739ed6738b8c |
| Ministral-3-14B | 57.8 | Not recorded | 6a0e919cf984739ed6738b8c |
| Gemini 3.1 Flash Lite | 53.2 | Not recorded | 6a0e919cf984739ed6738b8c |
| Qwen3.6-35B-A3B | 48.4 | Not recorded | 6a0e919cf984739ed6738b8c |
| Qwen3.6-35B-A3B | 40.1 | Not recorded | 6a0e919cf984739ed6738b8c |
| gpt-5.4-mini | 28.1 | Not recorded | 6a0e919cf984739ed6738b8c |
| Claude Haiku 4.5 | 27.9 | Not recorded | 6a0e919cf984739ed6738b8c |
| Claude Sonnet 4.6 | 11.2 | Not recorded | 6a0e919cf984739ed6738b8c |
| Ministral-3-14B | 7.8 | Not recorded | 6a0e919cf984739ed6738b8c |
| Claude Haiku 4.5 | 6.6 | Not recorded | 6a0e919cf984739ed6738b8c |

### gpt-5.4-mini

Utility Rate: 98.4. Result ID: 6a0ee565fbc373592c06f716.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"aware"}. This record does not assert independent reproduction.

### Gemma-4-26B-A4B-it

Utility Rate: 95.3. Result ID: 6a0ee565fbc373592c06f71c.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"aware"}. This record does not assert independent reproduction.

### Claude Sonnet 4.6

Utility Rate: 78.5. Result ID: 6a0ee565fbc373592c06f712.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"aware"}. This record does not assert independent reproduction.

### Gemma-4-26B-A4B-it

Utility Rate: 72.9. Result ID: 6a0ee565fbc373592c06f71d.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"blind"}. This record does not assert independent reproduction.

### gpt-oss-20B

Utility Rate: 71.4. Result ID: 6a0ee565fbc373592c06f720.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"aware"}. This record does not assert independent reproduction.

### Gemini 3.1 Flash Lite

Utility Rate: 69.8. Result ID: 6a0ee565fbc373592c06f718.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"aware"}. This record does not assert independent reproduction.

### gpt-oss-20B

Utility Rate: 60.3. Result ID: 6a0ee565fbc373592c06f721.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"blind"}. This record does not assert independent reproduction.

### Ministral-3-14B

Utility Rate: 57.8. Result ID: 6a0ee565fbc373592c06f71f.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"blind"}. This record does not assert independent reproduction.

### Gemini 3.1 Flash Lite

Utility Rate: 53.2. Result ID: 6a0ee565fbc373592c06f719.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"blind"}. This record does not assert independent reproduction.

### Qwen3.6-35B-A3B

Utility Rate: 48.4. Result ID: 6a0ee565fbc373592c06f71b.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"blind"}. This record does not assert independent reproduction.

### Qwen3.6-35B-A3B

Utility Rate: 40.1. Result ID: 6a0ee565fbc373592c06f71a.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"aware"}. This record does not assert independent reproduction.

### gpt-5.4-mini

Utility Rate: 28.1. Result ID: 6a0ee565fbc373592c06f717.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"blind"}. This record does not assert independent reproduction.

### Claude Haiku 4.5

Utility Rate: 27.9. Result ID: 6a0ee565fbc373592c06f714.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"aware"}. This record does not assert independent reproduction.

### Claude Sonnet 4.6

Utility Rate: 11.2. Result ID: 6a0ee565fbc373592c06f713.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"blind"}. This record does not assert independent reproduction.

### Ministral-3-14B

Utility Rate: 7.8. Result ID: 6a0ee565fbc373592c06f71e.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"open","Mode":"aware"}. This record does not assert independent reproduction.

### Claude Haiku 4.5

Utility Rate: 6.6. Result ID: 6a0ee565fbc373592c06f715.

[Reporting paper](<https://www.sota2.com/research/paper/6a0e919cf984739ed6738b8c.md>)

Recorded evaluation configuration: {"Tier":"frontier","Mode":"blind"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.