Sequence Understanding on Sequence Understanding Benchmark Average of 10 tasks
100Answer RateDeepSeek-V3
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DeepSeek-V3Protocol=Vanilla2026.04 | 100 | 45.7 | |
| DeepSeek-V3Protocol=CoT2026.04 | 100 | 44.5 | |
| DeepSeek-V3Protocol=ICL2026.04 | 100 | 45.2 | |
| DeepSeek-V3Protocol=SepSeq2026.04 | 100 | 50.9 | |
| Claude-3.7-SonnetProtocol=ICL2026.04 | 100 | 54.2 | |
| DeepSeek-R1Protocol=Vanilla2026.04 | 99.9 | 61.1 | |
| DeepSeek-R1Protocol=CoT2026.04 | 99.9 | 56.6 | |
| DeepSeek-R1Protocol=SepSeq2026.04 | 99.9 | 70.5 | |
| Claude-3.7-SonnetProtocol=CoT2026.04 | 99.9 | 57.6 | |
| Claude-3.7-SonnetProtocol=SepSeq2026.04 | 99.9 | 79.1 | |
| DeepSeek-R1Protocol=ICL2026.04 | 99.8 | 57.9 | |
| Claude-3.7-SonnetProtocol=Vanilla2026.04 | 99.8 | 57.3 | |
| GPT-4.1Protocol=ICL2026.04 | 99.8 | 60 | |
| GPT-4.1Protocol=Vanilla2026.04 | 99.6 | 61 | |
| GPT-4.1Protocol=CoT2026.04 | 99.6 | 60 | |
| GPT-4.1Protocol=SepSeq2026.04 | 99.5 | 82.7 | |
| GPT-4oProtocol=Vanilla2026.04 | 99 | 46.4 | |
| GPT-4oProtocol=ICL2026.04 | 98.6 | 44.5 | |
| GPT-4oProtocol=SepSeq2026.04 | 98.6 | 59.1 | |
| GPT-4oProtocol=CoT2026.04 | 98.5 | 51.5 | |
| AverageProtocol=SepSeq2026.04 | 92.8 | 68.9 | |
| AverageProtocol=Vanilla2026.04 | 90.5 | 51.6 | |
| AverageProtocol=CoT2026.04 | 90.3 | 48.9 | |
| Qwen3-30B-A3BProtocol=SepSeq2026.04 | 90.2 | 80.7 | |
| AverageProtocol=ICL2026.04 | 89.9 | 48.7 | |
| Qwen3-8BProtocol=SepSeq2026.04 | 87.3 | 69.6 | |
| Gemini-2.5-ProProtocol=SepSeq2026.04 | 84.6 | 79.1 | |
| Gemini-2.5-ProProtocol=Vanilla2026.04 | 83.4 | 58.9 | |
| Gemini-2.5-ProProtocol=ICL2026.04 | 82 | 56.9 | |
| Gemini-2.5-ProProtocol=CoT2026.04 | 81.8 | 48 | |
| Qwen3-30B-A3BProtocol=ICL2026.04 | 81 | 52.3 | |
| Qwen3-30B-A3BProtocol=CoT2026.04 | 80.9 | 52.7 | |
| Qwen3-30B-A3BProtocol=Vanilla2026.04 | 80.6 | 54.3 | |
| Qwen3-8BProtocol=CoT2026.04 | 79.5 | 40.3 | |
| Qwen3-8BProtocol=Vanilla2026.04 | 79.4 | 45.5 | |
| QwQ-32BProtocol=SepSeq2026.04 | 75.6 | 57.8 | |
| Qwen3-8BProtocol=ICL2026.04 | 75.2 | 38.5 | |
| QwQ-32BProtocol=Vanilla2026.04 | 72.7 | 34.2 | |
| QwQ-32BProtocol=ICL2026.04 | 72.7 | 28.5 | |
| QwQ-32BProtocol=CoT2026.04 | 72.2 | 28.9 |