Shift Cipher Decoding on Shift Cipher
98.4AccuracyZCoT
Evaluation results
| Method | Links | |
|---|---|---|
| ZCoTBackbone=GPT o3 mini (high), Setting=Zero-shot2026.05 | 98.4 | |
| STRATEGY-INDUCTBackbone=GPT o3 mini (high), Setting=Zero-shot2026.05 | 98.4 | |
| INDUCTBackbone=GPT o3 mini (high), Setting=Zero-shot2026.05 | 98.24 | |
| ZCoTBackbone=GPT o3 mini (medium), Setting=Zero-shot2026.05 | 97.44 | |
| STRATEGY-INDUCTBackbone=GPT o3 mini (medium), Setting=Zero-shot2026.05 | 97.44 | |
| STRATEGY-INDUCTBackbone=GPT o3 mini (low), Setting=Zero-shot2026.05 | 96.8 | |
| INDUCTBackbone=GPT o3 mini (low), Setting=Zero-shot2026.05 | 95.04 | |
| ZCoTBackbone=GPT o3 mini (low), Setting=Zero-shot2026.05 | 93.6 | |
| INDUCTBackbone=GPT o3 mini (medium), Setting=Zero-shot2026.05 | 88.48 | |
| STRATEGY-INDUCTBackbone=Gemini 2.0 Flash Thinking, Setting=Zero-shot2026.05 | 86.88 | |
| STRATEGY-INDUCTBackbone=GPT-4o, Setting=Zero-shot2026.05 | 84.8 | |
| INDUCTBackbone=Gemini 2.0 Flash Thinking, Setting=Zero-shot2026.05 | 81.76 | |
| SCoTBackbone=Gemini 2.0 Flash Thinking, Setting=Zero-shot2026.05 | 81.12 | |
| ZCoTBackbone=Gemini 2.0 Flash Thinking, Setting=Zero-shot2026.05 | 80 | |
| ZCoTBackbone=GPT-4o, Setting=Zero-shot2026.05 | 79.52 | |
| ZCoTBackbone=GPT-4o mini, Setting=Zero-shot2026.05 | 77.44 | |
| INDUCTBackbone=Gemini 2.0 Pro Exp, Setting=Zero-shot2026.05 | 77.12 | |
| INDUCTBackbone=GPT-4o mini, Setting=Zero-shot2026.05 | 76.16 | |
| SCoTBackbone=GPT-4o mini, Setting=Zero-shot2026.05 | 75.2 | |
| SCoTBackbone=GPT-4o, Setting=Zero-shot2026.05 | 75.2 | |
| INDUCTBackbone=GPT-4o, Setting=Zero-shot2026.05 | 73.76 | |
| SCoTBackbone=GPT o3 mini (low), Setting=Zero-shot2026.05 | 72.16 | |
| STRATEGY-INDUCTBackbone=Gemini 2.0 Pro Exp, Setting=Zero-shot2026.05 | 71.04 | |
| STRATEGY-INDUCTBackbone=Gemini 2.0 Flash, Setting=Zero-shot2026.05 | 67.04 | |
| STRATEGY-INDUCTBackbone=GPT-4o mini, Setting=Zero-shot2026.05 | 66.56 | |
| INDUCTBackbone=Llama 3.1 405B, Setting=Zero-shot2026.05 | 65.92 | |
| INDUCTBackbone=Gemini 2.0 Flash, Setting=Zero-shot2026.05 | 65.6 | |
| INDUCTBackbone=Gemini 1.5 Pro, Setting=Zero-shot2026.05 | 64.16 | |
| STRATEGY-INDUCTBackbone=Llama 3.1 405B, Setting=Zero-shot2026.05 | 61.44 | |
| STRATEGY-INDUCTBackbone=Gemini 1.5 Pro, Setting=Zero-shot2026.05 | 57.6 | |
| STRATEGY-INDUCTBackbone=Llama 3.3 70B, Setting=Zero-shot2026.05 | 56 | |
| ZCoTBackbone=Gemini 2.0 Pro Exp, Setting=Zero-shot2026.05 | 55.04 | |
| SCoTBackbone=Gemini 2.0 Pro Exp, Setting=Zero-shot2026.05 | 54.56 | |
| ZCoTBackbone=Gemini 2.0 Flash, Setting=Zero-shot2026.05 | 54.24 | |
| INDUCTBackbone=Llama 3.3 70B, Setting=Zero-shot2026.05 | 53.44 | |
| SCoTBackbone=Gemini 2.0 Flash, Setting=Zero-shot2026.05 | 53.44 | |
| INDUCTBackbone=Gemini 2.0 Flash Lite, Setting=Zero-shot2026.05 | 52 | |
| STRATEGY-INDUCTBackbone=Gemini 2.0 Flash Lite, Setting=Zero-shot2026.05 | 52 | |
| STRATEGY-INDUCTBackbone=Llama 3.1 70B, Setting=Zero-shot2026.05 | 50.24 | |
| INDUCTBackbone=Llama 3.1 70B, Setting=Zero-shot2026.05 | 47.84 | |
| SCoTBackbone=Mistral Large 2, Setting=Zero-shot2026.05 | 44.32 | |
| ZCoTBackbone=Llama 3.1 405B, Setting=Zero-shot2026.05 | 44.16 | |
| SCoTBackbone=Gemini 1.5 Pro, Setting=Zero-shot2026.05 | 41.28 | |
| STRATEGY-INDUCTBackbone=Mistral Large 2, Setting=Zero-shot2026.05 | 41.12 | |
| ZCoTBackbone=Mistral Large 2, Setting=Zero-shot2026.05 | 38.56 | |
| ZCoTBackbone=Gemini 2.0 Flash Lite, Setting=Zero-shot2026.05 | 36.16 | |
| SCoTBackbone=Llama 3.1 405B, Setting=Zero-shot2026.05 | 34.24 | |
| ZCoTBackbone=Llama 3.3 70B, Setting=Zero-shot2026.05 | 34.24 | |
| SCoTBackbone=Llama 3.1 70B, Setting=Zero-shot2026.05 | 32.16 | |
| SCoTBackbone=Llama 3.3 70B, Setting=Zero-shot2026.05 | 31.84 | |
| SCoTBackbone=GPT o3 mini (medium), Setting=Zero-shot2026.05 | 31.2 | |
| ZCoTBackbone=Llama 3.1 70B, Setting=Zero-shot2026.05 | 30.4 | |
| ZCoTBackbone=Gemini 1.5 Pro, Setting=Zero-shot2026.05 | 30.4 | |
| STRATEGY-INDUCTBackbone=Gemini 1.5 Flash, Setting=Zero-shot2026.05 | 26.4 | |
| SCoTBackbone=GPT o3 mini (high), Setting=Zero-shot2026.05 | 24.64 | |
| INDUCTBackbone=Mistral Large 2, Setting=Zero-shot2026.05 | 23.84 | |
| STRATEGY-INDUCTBackbone=Mistral Small 3, Setting=Zero-shot2026.05 | 21.44 | |
| INDUCTBackbone=Mistral Small 3, Setting=Zero-shot2026.05 | 20.48 | |
| SCoTBackbone=Gemini 2.0 Flash Lite, Setting=Zero-shot2026.05 | 19.68 | |
| ZCoTBackbone=Mistral Small 3, Setting=Zero-shot2026.05 | 18.24 | |
| SCoTBackbone=Mistral Small 3, Setting=Zero-shot2026.05 | 18.08 | |
| INDUCTBackbone=Gemini 1.5 Flash, Setting=Zero-shot2026.05 | 15.68 | |
| SCoTBackbone=Gemini 1.5 Flash, Setting=Zero-shot2026.05 | 13.76 | |
| ZCoTBackbone=Gemini 1.5 Flash, Setting=Zero-shot2026.05 | 6.88 | |
| SCoTBackbone=Gemini 1.5 Flash 8B, Setting=Zero-shot2026.05 | 4.16 | |
| INDUCTBackbone=Gemini 1.5 Flash 8B, Setting=Zero-shot2026.05 | 3.84 | |
| STRATEGY-INDUCTBackbone=Gemini 1.5 Flash 8B, Setting=Zero-shot2026.05 | 3.84 | |
| ZCoTBackbone=Gemini 1.5 Flash 8B, Setting=Zero-shot2026.05 | 3.52 | |
| INDUCTBackbone=Llama 3.1 8B, Setting=Zero-shot2026.05 | 2.56 | |
| STRATEGY-INDUCTBackbone=Llama 3.1 8B, Setting=Zero-shot2026.05 | 2.56 | |
| SCoTBackbone=Mistral Small 2, Setting=Zero-shot2026.05 | 2.08 | |
| ZCoTBackbone=Mistral Small 2, Setting=Zero-shot2026.05 | 1.76 | |
| STRATEGY-INDUCTBackbone=Mistral Small 2, Setting=Zero-shot2026.05 | 1.76 | |
| INDUCTBackbone=Mistral Small 2, Setting=Zero-shot2026.05 | 0.96 | |
| ZCoTBackbone=Llama 3.1 8B, Setting=Zero-shot2026.05 | 0.8 | |
| SCoTBackbone=Llama 3.1 8B, Setting=Zero-shot2026.05 | 0.64 | |
| STRATEGY-INDUCTBackbone=Mistral Nemo 12B, Setting=Zero-shot2026.05 | 0.48 | |
| SCoTBackbone=Mistral Nemo 12B, Setting=Zero-shot2026.05 | 0.16 | |
| ZCoTBackbone=Mistral Nemo 12B, Setting=Zero-shot2026.05 | 0 | |
| INDUCTBackbone=Mistral Nemo 12B, Setting=Zero-shot2026.05 | 0 |