IDS generation on Ishigaki-IDS-Bench
75.3IDSAudit PassIshigaki-IDS-14B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Ishigaki-IDS-14BModel group=Ours (scaling)2026.06 | 75.3 | 43.2 | 39.2 | |
| Ishigaki-IDS-32BModel group=Ours (scaling)2026.06 | 69.3 | 45.6 | 36.9 | |
| Ishigaki-IDS-8BModel group=Ours (released)2026.06 | 65.1 | 35.9 | 28.2 | |
| Claude Opus 4.5Model group=Closed LLM2026.06 | 33.1 | 42.2 | 24.9 | |
| GPT-5.5 (xhigh)Model group=Closed LLM2026.06 | 27.7 | 65.6 | 23.6 | |
| Gemini 3.1 ProModel group=Closed LLM2026.06 | 25.9 | 52.2 | 23.6 | |
| Kimi K2.6Model group=General open-weight LLM2026.06 | 22.3 | 48.4 | 20.9 | |
| Qwen3.5-397B-A17BModel group=General open-weight LLM2026.06 | 19.3 | 26.8 | 16.4 | |
| DeepSeek V4 ProModel group=General open-weight LLM2026.06 | 16.3 | 40 | 15 | |
| gpt-oss-120b (xhigh)Model group=General open-weight LLM2026.06 | 12 | 21.6 | 11.1 |