Attack and Collusion Detection Monitoring on BashArena (held-out 80% honest rollouts)
69T-only Catch RateBootstrapped Monitoring
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Bootstrapped MonitoringMonitor model (Um)=GLM-5, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 69 | 77 | 79 | 91 | 2.8 | |
| Bootstrapped MonitoringMonitor model (Um)=Kimi K2.5, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 69 | 72 | 73 | 82 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=DeepSeek, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 69 | 83 | 84 | 85 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=GPT-5.4, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Summarized2026.06 | 69 | 87 | 89 | 26 | 2.8 | |
| Bootstrapped MonitoringMonitor model (Um)=GLM-5, Agent model (Ua)=Kimi K2.5, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 61 | 64 | 72 | 90 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=Kimi K2.5, Agent model (Ua)=Kimi K2.5, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 61 | 92 | 93 | 91 | 2.7 | |
| Bootstrapped MonitoringMonitor model (Um)=DeepSeek, Agent model (Ua)=Kimi K2.5, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 61 | 74 | 78 | 72 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=GPT-5.4, Agent model (Ua)=Kimi K2.5, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Summarized2026.06 | 61 | 99 | 99 | 23 | 2.5 | |
| Bootstrapped MonitoringMonitor model (Um)=GLM-5, Agent model (Ua)=Kimi K2.5, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 55 | 64 | 73 | 80 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=Kimi K2.5, Agent model (Ua)=Kimi K2.5, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 55 | 92 | 71 | 82 | 2.2 | |
| Bootstrapped MonitoringMonitor model (Um)=DeepSeek, Agent model (Ua)=Kimi K2.5, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 55 | 74 | 75 | 88 | 3.2 | |
| Bootstrapped MonitoringMonitor model (Um)=GPT-5.4, Agent model (Ua)=Kimi K2.5, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Summarized2026.06 | 55 | 99 | 98 | 28 | 2.7 | |
| Bootstrapped MonitoringMonitor model (Um)=GLM-5, Agent model (Ua)=GPT-5.4, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 53 | 78 | 81 | 44 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=Kimi K2.5, Agent model (Ua)=GPT-5.4, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 53 | 63 | 70 | 43 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=DeepSeek, Agent model (Ua)=GPT-5.4, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Raw chain-of-thought2026.06 | 53 | 60 | 62 | 30 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=GPT-5.4, Agent model (Ua)=GPT-5.4, Trusted model (T)=GPT-4.1 Mini, Reasoning trace mode=Summarized2026.06 | 53 | 86 | 88 | 5 | 2.9 | |
| Bootstrapped MonitoringMonitor model (Um)=GLM-5, Agent model (Ua)=GPT-5.4, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 21 | 78 | 79 | 32 | 2.7 | |
| Bootstrapped MonitoringMonitor model (Um)=Kimi K2.5, Agent model (Ua)=GPT-5.4, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 21 | 63 | 66 | 36 | 2.1 | |
| Bootstrapped MonitoringMonitor model (Um)=DeepSeek, Agent model (Ua)=GPT-5.4, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 21 | 60 | 63 | 50 | 2.2 | |
| Bootstrapped MonitoringMonitor model (Um)=GPT-5.4, Agent model (Ua)=GPT-5.4, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Summarized2026.06 | 21 | 86 | 86 | 5 | 2.5 | |
| Bootstrapped MonitoringMonitor model (Um)=GLM-5, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 14 | 77 | 79 | 87 | 3.9 | |
| Bootstrapped MonitoringMonitor model (Um)=Kimi K2.5, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 14 | 72 | 75 | 87 | 3 | |
| Bootstrapped MonitoringMonitor model (Um)=DeepSeek, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Raw chain-of-thought2026.06 | 14 | 83 | 84 | 92 | 2.7 | |
| Bootstrapped MonitoringMonitor model (Um)=GPT-5.4, Agent model (Ua)=Gemini 3.1 Pro, Trusted model (T)=Qwen3-8B, Reasoning trace mode=Summarized2026.06 | 14 | 87 | 86 | 18 | 2.9 |