Task-solving Performance on BIG-Bench Hard (test)
68.3Causal JudgementBreeder
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Breederselection_strategy=highest development-set accuracy (oracle proxy)2025.10 | 68.3 | — | 89.8 | 74.5 | — | 74.6 | 68.4 | 93.2 | 77.8 | 72.4 | — | — | — | 90.8 | — | 92.7 | — | — | 69.8 | 90.3 | — | — | 85.9 | 60 | 92.7 | 96.3 | — | — | |
| OPROselection_strategy=highest development-set accuracy (oracle proxy)2025.10 | 68.2 | — | 91 | 73.4 | — | 72.8 | 56.9 | 93.2 | 77.4 | 71.8 | — | — | — | 88.2 | — | 91.9 | — | — | 68.7 | 88.8 | — | — | 83.3 | 66.2 | 92.4 | 93.8 | — | — | |
| APEselection_strategy=highest development-set accuracy (oracle proxy)2025.10 | 68 | — | 89.2 | 73 | — | 74.7 | 67 | 94 | 82.2 | 72.1 | — | — | — | 89.9 | — | 92.5 | — | — | 68.6 | 87.5 | — | — | 80.3 | 60 | 91.1 | 94.8 | — | — | |
| PDOselection_strategy=highest development-set accuracy (oracle proxy)2025.10 | 68 | — | 91.5 | 74 | — | 75.4 | 71.2 | 94.8 | 80.9 | 73.3 | — | — | — | 89 | — | 93.3 | — | — | 69.5 | 87.4 | — | — | 84.7 | 66.2 | 94.6 | 97.9 | — | — | |
| EvoPrompt(DE)-OPTS(TS)Prompt Strategy=EvoPrompt(DE), Selection Mechanism=OPTS(TS), Task-solving LLM=Llama-3-8B-Instruct2025.03 | 45.5 | 82.5 | 20.17 | 42.5 | 7.5 | 43.5 | 35.83 | 60.5 | 37.17 | 13 | 78.83 | 54.33 | 52.5 | 84.67 | 85.83 | 40.97 | 49.5 | 67.67 | 51.17 | 64.06 | 78.45 | 66 | 69.33 | 49.5 | 81.67 | 95 | 45.33 | 55.67 | |
| EvoPrompt(DE)-OPTS(APET)Prompt Strategy=EvoPrompt(DE), Selection Mechanism=OPTS(APET), Task-solving LLM=Llama-3-8B-Instruct2025.03 | 42.34 | 79.83 | 19.17 | 38.33 | 6.5 | 44.83 | 33 | 70 | 29 | 10.17 | 70.17 | 49.5 | 52.17 | 80.17 | 85.67 | 28.47 | 50 | 68.67 | 46.5 | 58.59 | 61.62 | 65.83 | 66.17 | 52.5 | 77.5 | 95 | 45.83 | 52.87 | |
| EvoPrompt(DE)Prompt Strategy=EvoPrompt(DE), Task-solving LLM=Llama-3-8B-Instruct2025.03 | 40.39 | 74.5 | 17.17 | 30 | 6.67 | 40.67 | 36 | 54.67 | 14.67 | 5.83 | 45.83 | 49.5 | 53.5 | 83.17 | 85.83 | 22.57 | 53 | 27.17 | 46.17 | 55.47 | 61.45 | 65.83 | 66.5 | 52 | 78.67 | 95 | 45.5 | 48.43 | |
| EvoPrompt(DE)-OPTS(US)Prompt Strategy=EvoPrompt(DE), Selection Mechanism=OPTS(US), Task-solving LLM=Llama-3-8B-Instruct2025.03 | 40.15 | 84 | 18.67 | 47.33 | 6.17 | 42.5 | 33 | 59.5 | 24.67 | 13.17 | 71.83 | 48 | 50.33 | 81.67 | 85 | 41.67 | 45.67 | 64.83 | 51.17 | 65.1 | 76.94 | 60.33 | 68.17 | 54.33 | 79.17 | 95 | 46.17 | 53.87 | |
| ManualPrompt Strategy=Manual Prompt, Task-solving LLM=Llama-3-8B-Instruct2025.03 | 2.19 | 54 | 14 | 19.5 | 6.5 | 29.5 | 16.5 | 53 | 12 | 5.5 | 44 | 40.5 | 53.5 | 84.5 | 87.5 | 26.04 | 21 | 18.5 | 12.5 | 24.22 | 51.52 | 57 | 67.5 | 47.5 | 80.5 | 93.5 | 44.5 | 39.52 | |
| APETPrompt Strategy=APET, Task-solving LLM=Llama-3-8B-Instruct2025.03 | 0 | 67.5 | 3 | 22.5 | 0 | 0 | 24.5 | 3.5 | 3.5 | 3 | 20.5 | 0 | 52 | 38.5 | 27.5 | 8.33 | 6.5 | 65.5 | 6.5 | 0 | 0 | 6 | 18 | 19.5 | 80.5 | 42 | 41 | 20.73 |