# Mobile Agent Safety and Capability Evaluation on Phone-Harm + Normal-150 (merged)

Source: [Mobile Agent Safety and Capability Evaluation on Phone-Harm + Normal-150 (merged)](<https://www.sota2.com/research/sota/mobile-agent-safety-and-capability-evaluation-on-phone-harm-plus-normal-150-merged>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-13T09:10:05.928Z

Evaluation of mobile agent safety and capability performance using the Phone-Harm + Normal-150 (merged) dataset, measured by HR, GAR, IF1, mHR, and OIR.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Mobile Agent Safety and Capability Evaluation |
| Dataset (including any stated split) | Phone-Harm + Normal-150 (merged) |
| Metric | Harm Rate (HR) |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 5 of 5 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Harm Rate (HR)](<https://www.sota2.com/research/sota/mobile-agent-safety-and-capability-evaluation-on-phone-harm-plus-normal-150-merged.md?metric=Harm%20Rate%20(HR)>): Lower is better; unit: SCALAR.

- [Goal Achievement Rate (GAR)](<https://www.sota2.com/research/sota/mobile-agent-safety-and-capability-evaluation-on-phone-harm-plus-normal-150-merged.md?metric=Goal%20Achievement%20Rate%20(GAR)>): Higher is better; unit: PERCENT.

- [Instruction Following Rate 1 (IF1)](<https://www.sota2.com/research/sota/mobile-agent-safety-and-capability-evaluation-on-phone-harm-plus-normal-150-merged.md?metric=Instruction%20Following%20Rate%201%20(IF1)>): Higher is better; unit: PERCENT.

- [Mean Harm Rate (mHR)](<https://www.sota2.com/research/sota/mobile-agent-safety-and-capability-evaluation-on-phone-harm-plus-normal-150-merged.md?metric=Mean%20Harm%20Rate%20(mHR)>): Lower is better; unit: SCALAR.

- [Overall Incident Rate (OIR)](<https://www.sota2.com/research/sota/mobile-agent-safety-and-capability-evaluation-on-phone-harm-plus-normal-150-merged.md?metric=Overall%20Incident%20Rate%20(OIR)>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Harm Rate (HR) | Result date | Report paper |
| --- | --- | --- | --- |
| GPT-5 | 0.2 | Not recorded | 69dc786b7d0172140ec789a2 |
| Gemini-3 | 0.42 | Not recorded | 69dc786b7d0172140ec789a2 |
| AutoGLM-VLM | 1.79 | Not recorded | 69dc786b7d0172140ec789a2 |
| CORA | 2.19 | Not recorded | 69dc786b7d0172140ec789a2 |
| UI-TARS-1.5 | 2.87 | Not recorded | 69dc786b7d0172140ec789a2 |

### GPT-5

Harm Rate (HR): 0.2. Result ID: 69dcb29e47f34ce924c3db00.

[Reporting paper](<https://www.sota2.com/research/paper/69dc786b7d0172140ec789a2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/695f39f053fdd5c361743ecb.md>)

Recorded evaluation configuration: {"Setting":"Base"}. This record does not assert independent reproduction.

### Gemini-3

Harm Rate (HR): 0.42. Result ID: 69dcb29e47f34ce924c3db01.

[Reporting paper](<https://www.sota2.com/research/paper/69dc786b7d0172140ec789a2.md>)

Recorded evaluation configuration: {"Setting":"Base"}. This record does not assert independent reproduction.

### AutoGLM-VLM

Harm Rate (HR): 1.79. Result ID: 69dcb29e47f34ce924c3db03.

[Reporting paper](<https://www.sota2.com/research/paper/69dc786b7d0172140ec789a2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75921e209131cea68b5e.md>)

Recorded evaluation configuration: {"Setting":"Base"}. This record does not assert independent reproduction.

### CORA

Harm Rate (HR): 2.19. Result ID: 69dcb29e47f34ce924c3db04.

[Reporting paper](<https://www.sota2.com/research/paper/69dc786b7d0172140ec789a2.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69dc786b7d0172140ec789a2.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### UI-TARS-1.5

Harm Rate (HR): 2.87. Result ID: 69dcb29e47f34ce924c3db02.

[Reporting paper](<https://www.sota2.com/research/paper/69dc786b7d0172140ec789a2.md>)

Recorded evaluation configuration: {"Setting":"Base"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
