Logic Reasoning on Tracking Shuffled Objects (BBH)
71.33AccuracyRole-Play Prompting
Evaluation Results
| Method | Links | |
|---|---|---|
| Role-Play PromptingModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 71.33 | |
| Persona SwitchModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 70.4 | |
| Top-pModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 64.93 | |
| MultinomialModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 64.67 | |
| Random SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 64.67 | |
| Persona SwitchModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 64.53 | |
| Role-Play PromptingModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 64.13 | |
| Top-kModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 61.33 | |
| Random SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 61.15 | |
| Self-discoverBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 60.03 | |
| GreedyModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 58.8 | |
| Low-gap SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 58.67 | |
| Low-gap SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 58.13 | |
| GreedyModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 53.47 | |
| MultinomialModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 51.33 | |
| Top-pModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 51.07 | |
| Top-kModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 46.53 | |
| IAP-mvBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 42.4 | |
| Prompt #1Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 40 | |
| IAP-ssBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 38.8 | |
| Persona SwitchModel=Gemma-2 (2B), Zero-shot=true2026.01 | 34.67 | |
| Random SelectionModel=Gemma-2 (2B), Zero-shot=true2026.01 | 32.44 | |
| MultinomialModel=Gemma-2 (2B), Zero-shot=true2026.01 | 32.27 | |
| GreedyModel=Gemma-2 (2B), Zero-shot=true2026.01 | 31.87 | |
| Low-gap SelectionModel=Gemma-2 (2B), Zero-shot=true2026.01 | 31.07 | |
| Top-kModel=Gemma-2 (2B), Zero-shot=true2026.01 | 30.13 | |
| Prompt #2Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 29.6 | |
| Top-pModel=Gemma-2 (2B), Zero-shot=true2026.01 | 29.07 | |
| Role-Play PromptingModel=Gemma-2 (2B), Zero-shot=true2026.01 | 28.4 | |
| OPPRBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 28 | |
| IAP-mvBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 25.6 | |
| Prompt #4Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 24.4 | |
| Prompt #6Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 24.4 | |
| Prompt #7Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 24 | |
| Self-discoverBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 24 | |
| IAP-ssBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 24 | |
| Prompt #3Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 23.2 | |
| AMV (#1-7)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 20.8 | |
| OPPRBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 20.8 | |
| OPROevaluations=30 prompt evaluations2026.05 | 20.5 | |
| ReElicitevaluations=30 prompt evaluations2026.05 | 20.4 | |
| Prompt #5Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 20 | |
| APEevaluations=30 prompt evaluations2026.05 | 19.6 | |
| TextGradevaluations=30 prompt evaluations2026.05 | 18.6 | |
| Prompt #6Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 17.6 | |
| Prompt #2Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 16.8 | |
| Prompt #7Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 16.8 | |
| Prompt #8Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 16.8 | |
| PromptBreederevaluations=30 prompt evaluations2026.05 | 16.3 | |
| Prompt #5Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 16 | |
| Prompt #4Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 15.6 | |
| Prompt #9Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 13.6 | |
| Prompt #1Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 13.2 | |
| AMV (all)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 12.6 | |
| Prompt #3Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.09 | 12.4 | |
| Prompt #9Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 10.8 | |
| AMV (#1-7)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 8.8 | |
| Prompt #8Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 2 | |
| AMV (all)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.09 | 1.44 |