BIG-bench Hard (BBH) Comprehensive Evaluation
83.2Boolean ExpressionsPaLM540B
Evaluation results
| Method | Links | |||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PaLM540Bprompting=3-shot2023.03 | 83.2 | 61 | 53.6 | 60.8 | 28.4 | 53.6 | 37.6 | 70.8 | 60.4 | 87.2 | 1.6 | 62.4 | 51.2 | 44.5 | 38 | 76 | 48.8 | 78.1 | 80.4 | 39.6 | 19.6 | 51.2 | 32 | 62.7 | 40.9 | 52.3 | — | |
| GPT-NeoXprompting=3-shot2023.03 | 71.2 | 52.41 | 45.6 | 40.8 | 26 | 52.8 | 8 | 92 | 30.93 | 86.4 | 0.4 | 45.2 | 21.2 | 33.56 | 26 | 54 | 20.4 | 62.36 | 53.2 | 21.2 | 24.53 | 52.4 | 5.2 | 51.63 | 27.84 | 40.25 | 45 | |
| BLOOM176Bprompting=3-shot2023.03 | 69.2 | 51.87 | 50 | 40.4 | 42 | 52.8 | 22.4 | 92 | 34 | 91.2 | 0 | 50 | 36.8 | 40.41 | 36.8 | 54.8 | 23.6 | 72.47 | 53.2 | 36.8 | 23.47 | 51.2 | 7.6 | 54.96 | 33.95 | 44.91 | 75 | |
| BLOOMBERGGPTprompting=3-shot2023.03 | 62.4 | 49.73 | 54.8 | 34 | 15.6 | 50.8 | 15.2 | 92 | 34.53 | 90.4 | 1.2 | 42 | 33.2 | 37.67 | 34.8 | 56 | 20 | 69.66 | 62.8 | 29.2 | 25.33 | 49.2 | 4.8 | 54.39 | 28.42 | 41.97 | 57 | |
| OPT66Bprompting=3-shot2023.03 | 48.4 | 51.87 | 49.6 | 40.4 | 14.8 | 54 | 11.6 | 91.6 | 31.87 | 91.2 | 0.4 | 42 | 26 | 28.08 | 31.2 | 52.8 | 16.4 | 69.66 | 54.4 | 23.6 | 24 | 54 | 2.4 | 52.6 | 25.37 | 39.58 | 39 |