# NLU and Reasoning on BIG-bench (unseen)

Source: [NLU and Reasoning on BIG-bench (unseen)](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:47:55.621Z

Evaluation of model performance across a specific selection of 14 tasks within the BIG-bench suite, covering knowledge retrieval, logical reasoning, and conceptual understanding, resulting in a composite average score.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Various NLP tasks (NLU and Reasoning) |
| Dataset (including any stated split) | BIG-bench (unseen) |
| Metric | Known Unknowns Score |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Known Unknowns Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Known%20Unknowns%20Score>): Higher is better; unit: PERCENT.

- [Logic Grid Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Logic%20Grid%20Score>): Higher is better; unit: PERCENT.

- [Strategy Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Strategy%20Score>): Higher is better; unit: PERCENT.

- [Hindu Knowledge Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Hindu%20Knowledge%20Score>): Higher is better; unit: PERCENT.

- [Movie Domain Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Movie%20Domain%20Score>): Higher is better; unit: PERCENT.

- [Code Domain Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Code%20Domain%20Score>): Higher is better; unit: PERCENT.

- [Concept Understanding Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Concept%20Understanding%20Score>): Higher is better; unit: PERCENT.

- [Language Proficiency Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Language%20Proficiency%20Score>): Higher is better; unit: PERCENT.

- [Vitamin Task Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Vitamin%20Task%20Score>): Higher is better; unit: PERCENT.

- [Syllogism Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Syllogism%20Score>): Higher is better; unit: PERCENT.

- [Misconception Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Misconception%20Score>): Higher is better; unit: PERCENT.

- [Logical Reasoning Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Logical%20Reasoning%20Score>): Higher is better; unit: PERCENT.

- [Winowhy Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Winowhy%20Score>): Higher is better; unit: PERCENT.

- [BIG-bench Average Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=BIG-bench%20Average%20Score>): Higher is better; unit: PERCENT.

- [Novel Concept Score](<https://www.sota2.com/research/sota/nlu-and-reasoning-on-big-bench-unseen.md?metric=Novel%20Concept%20Score>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Known Unknowns Score | Result date | Report paper |
| --- | --- | --- | --- |
| FLIPPED | 86.96 | Not recorded | 68fa6fa11e209131cea46c90 |
| FLIPPED | 71.74 | Not recorded | 68fa6fa11e209131cea46c90 |
| PALM | 67.39 | Not recorded | 68fa6fa11e209131cea46c90 |
| T0 | 65.22 | Not recorded | 68fa6e7d1e209131cea4038a |
| DIRECT | 63.04 | Not recorded | 68fa6fa11e209131cea46c90 |
| GPT-3 | 60.87 | Not recorded | 68fa6e7d1e209131cea4038a |
| GPT-3 | 60.87 | Not recorded | 68fa6fa11e209131cea46c90 |
| Cos PE | 58.7 | Not recorded | 68fa6e7d1e209131cea4038a |
| T0 | 58.7 | Not recorded | 68fa6fa11e209131cea46c90 |
| PaLM | 56.52 | Not recorded | 68fa6e7d1e209131cea4038a |
| PALM | 56.52 | Not recorded | 68fa6fa11e209131cea46c90 |
| CHANNEL | 52.17 | Not recorded | 68fa6fa11e209131cea46c90 |
| GPT-3 | 50 | Not recorded | 68fa6fa11e209131cea46c90 |
| T0 | 47.83 | Not recorded | 68fa6e7d1e209131cea4038a |
| T0 | 47.83 | Not recorded | 68fa6fa11e209131cea46c90 |

### FLIPPED

Known Unknowns Score: 86.96. Result ID: 6990f79b3a6c31484904abf7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Implementation](<https://www.sota2.com/github.com/seonghyeonye/Flipped-Learning>)

Recorded evaluation configuration: {"Parameters":"11B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

### FLIPPED

Known Unknowns Score: 71.74. Result ID: 6990f79b3a6c31484904abf5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Implementation](<https://www.sota2.com/github.com/seonghyeonye/Flipped-Learning>)

Recorded evaluation configuration: {"Parameters":"3B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

### PALM

Known Unknowns Score: 67.39. Result ID: 6990f79b3a6c31484904abfb.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6d931e209131cea3abf2.md>)

Recorded evaluation configuration: {"Parameters":"540B","Setting":"Few-shot","Shots":1}. This record does not assert independent reproduction.

### T0

Known Unknowns Score: 65.22. Result ID: 699025073a6c314849fe6b83.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6e7d1e209131cea4038a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c131e209131cea31f71.md>)

Recorded evaluation configuration: {"Parameters":"11B","Training":"Instruction Tuning"}. This record does not assert independent reproduction.

### DIRECT

Known Unknowns Score: 63.04. Result ID: 6990f79b3a6c31484904abf3.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c131e209131cea31f71.md>)

Recorded evaluation configuration: {"Parameters":"3B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

### GPT-3

Known Unknowns Score: 60.87. Result ID: 699025073a6c314849fe6b84.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6e7d1e209131cea4038a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa680c1e209131cea19d53.md>)

[Implementation](<https://github.com/openai/gpt-3/blob/master/overlap_frequency.md>)

Recorded evaluation configuration: {"Parameters":"175B"}. This record does not assert independent reproduction.

### GPT-3

Known Unknowns Score: 60.87. Result ID: 6990f79b3a6c31484904abf8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa680c1e209131cea19d53.md>)

[Implementation](<https://github.com/openai/gpt-3/blob/master/overlap_frequency.md>)

Recorded evaluation configuration: {"Parameters":"175B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

### Cos PE

Known Unknowns Score: 58.7. Result ID: 699025073a6c314849fe6b82.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6e7d1e209131cea4038a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e7d1e209131cea4038a.md>)

[Implementation](<https://github.com/joeljang/ELM>)

Recorded evaluation configuration: {"Parameters":"3B","Method Variant":"Prompt Expert","Expert Backbone":"T5","Training Task":"COSMOS-QA"}. This record does not assert independent reproduction.

### T0

Known Unknowns Score: 58.7. Result ID: 6990f79b3a6c31484904abf6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c131e209131cea31f71.md>)

Recorded evaluation configuration: {"Parameters":"11B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

### PaLM

Known Unknowns Score: 56.52. Result ID: 699025073a6c314849fe6b85.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6e7d1e209131cea4038a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6d931e209131cea3abf2.md>)

Recorded evaluation configuration: {"Parameters":"540B"}. This record does not assert independent reproduction.

### PALM

Known Unknowns Score: 56.52. Result ID: 6990f79b3a6c31484904abf9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6d931e209131cea3abf2.md>)

Recorded evaluation configuration: {"Parameters":"540B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

### CHANNEL

Known Unknowns Score: 52.17. Result ID: 6990f79b3a6c31484904abf4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c7c1e209131cea34419.md>)

Recorded evaluation configuration: {"Parameters":"3B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

### GPT-3

Known Unknowns Score: 50. Result ID: 6990f79b3a6c31484904abfa.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa680c1e209131cea19d53.md>)

[Implementation](<https://github.com/openai/gpt-3/blob/master/overlap_frequency.md>)

Recorded evaluation configuration: {"Parameters":"175B","Setting":"Few-shot","Shots":3}. This record does not assert independent reproduction.

### T0

Known Unknowns Score: 47.83. Result ID: 699025073a6c314849fe6b81.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6e7d1e209131cea4038a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c131e209131cea31f71.md>)

Recorded evaluation configuration: {"Parameters":"3B","Training":"Instruction Tuning"}. This record does not assert independent reproduction.

### T0

Known Unknowns Score: 47.83. Result ID: 6990f79b3a6c31484904abf2.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6fa11e209131cea46c90.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c131e209131cea31f71.md>)

Recorded evaluation configuration: {"Parameters":"3B","Setting":"Zero-shot","Shots":0}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
