Systemic Risk Detection on Safe-OS
100Normal CountAgentMonitor
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| AgentMonitorFramework Category=Guardrail-based, Backbone=GPT-4o2025.02 | 100 | 46.7 | 39.1 | 85 | |
| LLaMA-Guard 3Framework Category=Guardrail-based2025.02 | 100 | 55.2 | 100 | 100 | |
| AGrailFramework Category=Guardrail-based, Backbone=Claude-3.52025.02 | 95.6 | 3.8 | 0 | 5 | |
| AGrailFramework Category=Guardrail-based, Backbone=GPT-4o2025.02 | 95.6 | 4 | 0 | 10 | |
| ToolEmuFramework Category=Guardrail-based, Backbone=Claude-3.52025.02 | 57.7 | 4.2 | 100 | 35 | |
| GPT-4oFramework Category=Model-based, Shots=0-Shot2025.02 | 52.4 | 7.7 | 61.9 | 15 | |
| Claude-3.5Framework Category=Model-based, Shots=0-Shot2025.02 | 50 | 0 | 14.3 | 20 |