Agent Behavioral Safety and Helpfulness Evaluation on ToolEmu
97.9Safety RateThought-Aligner-7B
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Thought-Aligner-7BCore LLM=o3(AzureOpenAI)2025.05 | 97.9 | 2.91 | 14.6 | 0.49 | |
| Thought-Aligner-1.5BCore LLM=o3(AzureOpenAI)2025.05 | 97.2 | 2.93 | 12.5 | 0.4 | |
| GuardAgentCore LLM=o3(AzureOpenAI)2025.05 | 96.2 | 2.92 | 9 | 0.28 | |
| Self-ReflectionCore LLM=o3(AzureOpenAI)2025.05 | 95.8 | 2.89 | 7.6 | 0.23 | |
| Thought-Aligner-7BCore LLM=GPT-4.12025.05 | 95.2 | 2.9 | 18.8 | 0.61 | |
| AthenaCore LLM=o3(AzureOpenAI)2025.05 | 95.1 | 2.87 | 26 | 0.81 | |
| Thought-Aligner-7BCore LLM=Claude-Sonnet-42025.05 | 95.1 | 2.73 | 44.4 | 1.25 | |
| Thought-Aligner-7BCore LLM=Qwen3-235B-A22B2025.05 | 95.1 | 2.68 | 43.1 | 1.33 | |
| ShieldAgentCore LLM=o3(AzureOpenAI)2025.05 | 94 | 2.54 | 8.3 | 0.28 | |
| Thought-Aligner-1.5BCore LLM=Qwen3-235B-A22B2025.05 | 93.8 | 2.6 | 45.1 | 1.28 | |
| Thought-Aligner-1.5BCore LLM=GPT-4.12025.05 | 93.1 | 2.87 | 21.5 | 0.95 | |
| Thought-Aligner-7BCore LLM=Llama-3.3-70B2025.05 | 93.1 | 2.47 | 39.6 | 1.24 | |
| Thought-Aligner-1.5BCore LLM=Llama-3.3-70B2025.05 | 92.7 | 2.41 | 42.4 | 1.28 | |
| Thought-Aligner-7BCore LLM=DeepSeek-V32025.05 | 92.2 | 2.78 | 37.5 | 1.17 | |
| Thought-Aligner-1.5BCore LLM=Claude-Sonnet-42025.05 | 91.7 | 2.74 | 42.4 | 1.3 | |
| Thought-Aligner-1.5BCore LLM=DeepSeek-V32025.05 | 91.5 | 2.79 | 31.3 | 1 | |
| GuardAgentCore LLM=GPT-4.12025.05 | 84.7 | 2.53 | 16 | 0.51 | |
| GuardAgentCore LLM=Claude-Sonnet-42025.05 | 84.7 | 2.53 | 22.2 | 0.7 | |
| AthenaCore LLM=GPT-4.12025.05 | 80.6 | 2.42 | 38.2 | 1.15 | |
| GuardAgentCore LLM=DeepSeek-V32025.05 | 80.6 | 2.46 | 14.6 | 0.51 | |
| AthenaCore LLM=Claude-Sonnet-42025.05 | 76.4 | 2.35 | 48.6 | 1.44 | |
| Self-ReflectionCore LLM=DeepSeek-V32025.05 | 75.7 | 2.37 | 13.2 | 0.44 | |
| Self-ReflectionCore LLM=GPT-4.12025.05 | 73.6 | 2.24 | 16.7 | 0.56 | |
| Self-ReflectionCore LLM=Llama-3.3-70B2025.05 | 73.6 | 2.24 | 42.4 | 1.13 | |
| Self-ReflectionCore LLM=Claude-Sonnet-42025.05 | 70.8 | 2.22 | 32.6 | 1.01 | |
| GuardAgentCore LLM=Qwen3-235B-A22B2025.05 | 70.8 | 2.21 | 39.6 | 1.12 | |
| No DefenseCore LLM=o3(AzureOpenAI)2025.05 | 69.4 | 2.07 | 3.4 | 0.1 | |
| GuardAgentCore LLM=Llama-3.3-70B2025.05 | 69.4 | 2.13 | 23.6 | 0.86 | |
| ShieldAgentCore LLM=Claude-Sonnet-42025.05 | 68.8 | 2.01 | 33.3 | 1.07 | |
| AthenaCore LLM=DeepSeek-V32025.05 | 67.4 | 2.06 | 37.5 | 1.15 | |
| ShieldAgentCore LLM=Llama-3.3-70B2025.05 | 65.3 | 1.76 | 38.2 | 1.19 | |
| ShieldAgentCore LLM=DeepSeek-V32025.05 | 62.5 | 1.81 | 29.9 | 0.98 | |
| No DefenseCore LLM=Claude-Sonnet-42025.05 | 61.8 | 1.83 | 35.4 | 1.05 | |
| ShieldAgentCore LLM=Qwen3-235B-A22B2025.05 | 61.8 | 1.74 | 40.3 | 1.31 | |
| Self-ReflectionCore LLM=Qwen3-235B-A22B2025.05 | 58.3 | 1.78 | 43.8 | 1.21 | |
| ShieldAgentCore LLM=GPT-4.12025.05 | 56.9 | 1.71 | 23.6 | 0.82 | |
| AthenaCore LLM=Qwen3-235B-A22B2025.05 | 56.3 | 1.8 | 22.2 | 0.79 | |
| AthenaCore LLM=Llama-3.3-70B2025.05 | 56.3 | 1.74 | 31.3 | 0.94 | |
| No DefenseCore LLM=DeepSeek-V32025.05 | 52.8 | 1.62 | 31.9 | 1.03 | |
| No DefenseCore LLM=Llama-3.3-70B2025.05 | 51.4 | 1.56 | 36.1 | 1.21 | |
| No DefenseCore LLM=Qwen3-235B-A22B2025.05 | 50.7 | 1.52 | 37.5 | 1.12 | |
| No DefenseCore LLM=GPT-4.12025.05 | 43.1 | 1.51 | 24.3 | 0.87 |