Trace-level safety monitoring on ImpossibleBench
95.7ROC-AUCMeerkat
Evaluation Results
| Method | Links | |
|---|---|---|
| MeerkatTPC=100, Model=GPT-5.4m2026.04 | 95.7 | |
| MeerkatTPC=100, Model=Qwen3.52026.04 | 95.1 | |
| BufferTPC=100, Model=GPT-5.4m2026.04 | 94.9 | |
| BufferTPC=100, Model=Qwen3.52026.04 | 94.8 | |
| MeerkatTPC=50, Model=GPT-5.4m2026.04 | 94.4 | |
| MonitorTPC=100, Model=GPT-5.4m2026.04 | 94.3 | |
| BayesianTPC=100, Model=Qwen3.52026.04 | 93.5 | |
| BayesianTPC=100, Model=GPT-5.4m2026.04 | 93.5 | |
| MonitorTPC=50, Model=GPT-5.4m2026.04 | 93.3 | |
| MeerkatTPC=50, Model=Qwen3.52026.04 | 92.8 | |
| MonitorTPC=10, Model=GPT-5.4m2026.04 | 92.4 | |
| MeerkatTPC=25, Model=GPT-5.4m2026.04 | 92.4 | |
| BufferTPC=50, Model=GPT-5.4m2026.04 | 91.5 | |
| BufferTPC=50, Model=Qwen3.52026.04 | 91.4 | |
| MonitorTPC=25, Model=GPT-5.4m2026.04 | 90 | |
| MeerkatTPC=10, Model=Qwen3.52026.04 | 89.8 | |
| MeerkatTPC=10, Model=GPT-5.4m2026.04 | 89.8 | |
| MonitorTPC=100, Model=Qwen3.52026.04 | 89.3 | |
| BayesianTPC=50, Model=Qwen3.52026.04 | 88 | |
| BayesianTPC=50, Model=GPT-5.4m2026.04 | 88 | |
| BayesianTPC=10, Model=Qwen3.52026.04 | 87.8 | |
| BayesianTPC=10, Model=GPT-5.4m2026.04 | 87.8 | |
| MeerkatTPC=25, Model=Qwen3.52026.04 | 87.6 | |
| BufferTPC=10, Model=GPT-5.4m2026.04 | 87.3 | |
| BufferTPC=10, Model=Qwen3.52026.04 | 87.1 | |
| MonitorTPC=50, Model=Qwen3.52026.04 | 87 | |
| BufferTPC=25, Model=Qwen3.52026.04 | 85.5 | |
| BufferTPC=25, Model=GPT-5.4m2026.04 | 85.5 | |
| BayesianTPC=25, Model=Qwen3.52026.04 | 84.8 | |
| BayesianTPC=25, Model=GPT-5.4m2026.04 | 84.8 | |
| MonitorTPC=10, Model=Qwen3.52026.04 | 80 | |
| Naive AgentTPC=50, Model=Qwen3.52026.04 | 78.9 | |
| MonitorTPC=25, Model=Qwen3.52026.04 | 78 | |
| Naive AgentTPC=100, Model=GPT-5.4m2026.04 | 74.2 | |
| Naive AgentTPC=10, Model=Qwen3.52026.04 | 73.8 | |
| Naive AgentTPC=25, Model=Qwen3.52026.04 | 71.8 | |
| Naive AgentTPC=100, Model=Qwen3.52026.04 | 70 | |
| Naive AgentTPC=50, Model=GPT-5.4m2026.04 | 61.9 | |
| Naive AgentTPC=10, Model=GPT-5.4m2026.04 | 50 | |
| Naive AgentTPC=25, Model=GPT-5.4m2026.04 | 50 |