Task-specific Risk Detection on Mind2Web-SC (test)
0.99LPAGPT-4o
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GPT-4oType=Model-based, Shots=1-shot2025.02 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | |
| AGrailType=Guardrail-based, Backbone=GPT-4o2025.02 | 0.984 | 0.99 | 0.98 | 0.984 | 0.947 | |
| GPT-4oType=Model-based, Shots=0-shot2025.02 | 0.96 | 0.969 | 0.949 | 0.959 | 0.78 | |
| Claude-3.5Type=Model-based, Shots=1-shot2025.02 | 0.943 | 0.898 | 1 | 0.946 | 0.989 | |
| AGrailType=Guardrail-based, Backbone=Claude-3.52025.02 | 0.94 | 0.914 | 0.97 | 0.941 | 0.958 | |
| Claude-3.5Type=Model-based, Shots=0-shot2025.02 | 0.933 | 0.892 | 1 | 0.943 | 0.99 | |
| GuardAgentType=Guardrail-based, Backbone=GPT-42025.02 | 0.9 | 1 | 0.8 | 0.89 | 0.9 | |
| AgentMonitorType=Guardrail-based, Backbone=GPT-4o2025.02 | 0.725 | 0.792 | 0.61 | 0.689 | 0.885 | |
| LLaMA-Guard3Type=Guardrail-based2025.02 | 0.56 | 0.93 | 0.13 | 0.23 | — |