List your company
ResearchFormerly WizWand

Logic Reasoning on Tracking Shuffled Objects (BBH)

71.33AccuracyRole-Play Prompting
-1.355617.514736.38555.2553Sep 30, 2024Jan 7, 2025Apr 16, 2025Jul 24, 2025Oct 31, 2025Feb 7, 2026May 18, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Role-Play PromptingModel=LLaMA-3.1 (8B), Zero-shot=true2026.0171.33
Persona SwitchModel=LLaMA-3.1 (8B), Zero-shot=true2026.0170.4
Top-pModel=LLaMA-3.1 (8B), Zero-shot=true2026.0164.93
MultinomialModel=LLaMA-3.1 (8B), Zero-shot=true2026.0164.67
Random SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.0164.67
Persona SwitchModel=LLaMA-3.2 (3B), Zero-shot=true2026.0164.53
Role-Play PromptingModel=LLaMA-3.2 (3B), Zero-shot=true2026.0164.13
Top-kModel=LLaMA-3.1 (8B), Zero-shot=true2026.0161.33
Random SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.0161.15
Self-discoverBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0960.03
GreedyModel=LLaMA-3.1 (8B), Zero-shot=true2026.0158.8
Low-gap SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.0158.67
Low-gap SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.0158.13
GreedyModel=LLaMA-3.2 (3B), Zero-shot=true2026.0153.47
MultinomialModel=LLaMA-3.2 (3B), Zero-shot=true2026.0151.33
Top-pModel=LLaMA-3.2 (3B), Zero-shot=true2026.0151.07
Top-kModel=LLaMA-3.2 (3B), Zero-shot=true2026.0146.53
IAP-mvBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0942.4
Prompt #1Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0940
IAP-ssBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0938.8
Persona SwitchModel=Gemma-2 (2B), Zero-shot=true2026.0134.67
Random SelectionModel=Gemma-2 (2B), Zero-shot=true2026.0132.44
MultinomialModel=Gemma-2 (2B), Zero-shot=true2026.0132.27
GreedyModel=Gemma-2 (2B), Zero-shot=true2026.0131.87
Low-gap SelectionModel=Gemma-2 (2B), Zero-shot=true2026.0131.07
Top-kModel=Gemma-2 (2B), Zero-shot=true2026.0130.13
Prompt #2Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0929.6
Top-pModel=Gemma-2 (2B), Zero-shot=true2026.0129.07
Role-Play PromptingModel=Gemma-2 (2B), Zero-shot=true2026.0128.4
OPPRBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0928
IAP-mvBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0925.6
Prompt #4Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0924.4
Prompt #6Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0924.4
Prompt #7Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0924
Self-discoverBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0924
IAP-ssBackbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0924
Prompt #3Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0923.2
AMV (#1-7)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0920.8
OPPRBackbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0920.8
OPROevaluations=30 prompt evaluations2026.0520.5
ReElicitevaluations=30 prompt evaluations2026.0520.4
Prompt #5Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0920
APEevaluations=30 prompt evaluations2026.0519.6
TextGradevaluations=30 prompt evaluations2026.0518.6
Prompt #6Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0917.6
Prompt #2Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0916.8
Prompt #7Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0916.8
Prompt #8Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0916.8
PromptBreederevaluations=30 prompt evaluations2026.0516.3
Prompt #5Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0916
Prompt #4Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0915.6
Prompt #9Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0913.6
Prompt #1Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0913.2
AMV (all)Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0912.6
Prompt #3Backbone=LLaMA-3-8B-Instruct, Zero-shot CoT=true2024.0912.4
Prompt #9Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.0910.8
AMV (#1-7)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.098.8
Prompt #8Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.092
AMV (all)Backbone=Qwen-14B-Chat, Zero-shot CoT=true2024.091.44