Attacker effectiveness on Chinese scam dialogues (test)
10.1Success Ratedeepseek-r1
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| deepseek-r1TOTAL=11202026.01 | 10.1 | 62.1 | 7.7 | 20.1 | |
| claude-sonnet-4-5-thinkingTOTAL=11202026.01 | 9.8 | 66.4 | 14 | 9.7 | |
| grok-4TOTAL=11202026.01 | 3.7 | 36.6 | 8.6 | 51.1 | |
| glm-4.6-thinkingTOTAL=11192026.01 | 3.4 | 26.8 | 17.2 | 52.6 | |
| SparkDesk-v3.5TOTAL=11202026.01 | 2.9 | 45 | 41.1 | 11 | |
| llama-3.1-405b-instructTOTAL=11162026.01 | 2.4 | 12.4 | 52.5 | 32.7 | |
| ERNIE-4.0-8KTOTAL=11202026.01 | 0.4 | 24.1 | 53.2 | 22.2 | |
| qwen-max-latestTOTAL=11202026.01 | 0.3 | 29.6 | 42.7 | 27.5 |