# Natural Language Understanding on GLUE 1.0 (dev)

Source: [Natural Language Understanding on GLUE 1.0 (dev)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T06:03:31.035Z

Evaluation of general language understanding capabilities across the constituent tasks of the GLUE 1.0 benchmark suite, specifically reporting results on the development (dev) split.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Natural Language Understanding |
| Dataset (including any stated split) | GLUE 1.0 (dev) |
| Metric | SST-2 (Acc) |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 18 of 18 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [SST-2 (Acc)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=SST-2%20(Acc)>): Higher is better; unit: PERCENT.

- [CoLA (MCC)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=CoLA%20(MCC)>): Higher is better; unit: PERCENT.

- [MRPC (Acc)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=MRPC%20(Acc)>): Higher is better; unit: PERCENT.

- [QNLI (Acc)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=QNLI%20(Acc)>): Higher is better; unit: PERCENT.

- [QQP (Acc)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=QQP%20(Acc)>): Higher is better; unit: PERCENT.

- [RTE (Acc)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=RTE%20(Acc)>): Higher is better; unit: PERCENT.

- [STS-B (PCC)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=STS-B%20(PCC)>): Higher is better; unit: PERCENT.

- [MNLI (Acc)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-1-0-dev.md?metric=MNLI%20(Acc)>): Higher is better; unit: PERCENT.

## Recorded results

| Method | SST-2 (Acc) | Result date | Report paper |
| --- | --- | --- | --- |
| RoBERTa | 95.3 | Not recorded | 68fa6c001e209131cea3193a |
| XLNet | 94.4 | Not recorded | 68fa6c001e209131cea3193a |
| BERT | 93.4 | Not recorded | 68fa6c001e209131cea3193a |
| PERT | 93.4 | Not recorded | 68fa6c001e209131cea3193a |
| BERT | 93.2 | Not recorded | 68fa6c001e209131cea3193a |
| BERT | 92.7 | Not recorded | 68fa6c001e209131cea3193a |
| BERT | 92.6 | Not recorded | 68fa6c001e209131cea3193a |
| XLNet | 92.6 | Not recorded | 68fa6c001e209131cea3193a |
| RoBERTa | 92.5 | Not recorded | 68fa6c001e209131cea3193a |
| BERT baseline | 92.36 | Not recorded | 68fa6b7f1e209131cea2e9dc |
| Q8BERT | 92.24 | Not recorded | 68fa6b7f1e209131cea2e9dc |
| PERT | 92 | Not recorded | 68fa6c001e209131cea3193a |
| DQ BERT | 91.04 | Not recorded | 68fa6b7f1e209131cea2e9dc |
| ALBERT | 90.6 | Not recorded | 68fa6c001e209131cea3193a |
| ALBERT | 89.4 | Not recorded | 68fa6c001e209131cea3193a |

### RoBERTa

SST-2 (Acc): 95.3. Result ID: 698eb866ffc3901de0a01403.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66131e209131cea0e18b.md>)

[Implementation](<https://github.com/pytorch/fairseq>)

Recorded evaluation configuration: {"Scale":"large"}. This record does not assert independent reproduction.

### XLNet

SST-2 (Acc): 94.4. Result ID: 698eb866ffc3901de0a01404.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66e61e209131cea1300d.md>)

[Implementation](<https://github.com/zihangdai/xlnet>)

Recorded evaluation configuration: {"Scale":"large"}. This record does not assert independent reproduction.

### BERT

SST-2 (Acc): 93.4. Result ID: 698eb866ffc3901de0a01402.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65c31e209131cea0c4ed.md>)

[Implementation](<https://github.com/google-research/bert>)

Recorded evaluation configuration: {"Scale":"large","variant":"wwm","reproduced":true}. This record does not assert independent reproduction.

### PERT

SST-2 (Acc): 93.4. Result ID: 698eb866ffc3901de0a01406.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Implementation](<https://github.com/ymcui/PERT>)

Recorded evaluation configuration: {"Scale":"large"}. This record does not assert independent reproduction.

### BERT

SST-2 (Acc): 93.2. Result ID: 698eb866ffc3901de0a01401.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65c31e209131cea0c4ed.md>)

[Implementation](<https://github.com/google-research/bert>)

Recorded evaluation configuration: {"Scale":"large"}. This record does not assert independent reproduction.

### BERT

SST-2 (Acc): 92.7. Result ID: 698eb866ffc3901de0a013fb.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65c31e209131cea0c4ed.md>)

[Implementation](<https://github.com/google-research/bert>)

Recorded evaluation configuration: {"Scale":"base"}. This record does not assert independent reproduction.

### BERT

SST-2 (Acc): 92.6. Result ID: 698eb866ffc3901de0a013fc.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65c31e209131cea0c4ed.md>)

[Implementation](<https://github.com/google-research/bert>)

Recorded evaluation configuration: {"Scale":"base","variant":"+"}. This record does not assert independent reproduction.

### XLNet

SST-2 (Acc): 92.6. Result ID: 698eb866ffc3901de0a013fe.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66e61e209131cea1300d.md>)

[Implementation](<https://github.com/zihangdai/xlnet>)

Recorded evaluation configuration: {"Scale":"base"}. This record does not assert independent reproduction.

### RoBERTa

SST-2 (Acc): 92.5. Result ID: 698eb866ffc3901de0a013fd.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66131e209131cea0e18b.md>)

[Implementation](<https://github.com/pytorch/fairseq>)

Recorded evaluation configuration: {"Scale":"base"}. This record does not assert independent reproduction.

### BERT baseline

SST-2 (Acc): 92.36. Result ID: 698e90ecffc3901de09eb86a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6b7f1e209131cea2e9dc.md>)

Recorded evaluation configuration: {"Architecture":"BERT-Base","Precision":"FP32","Quantization Method":"None"}. This record does not assert independent reproduction.

### Q8BERT

SST-2 (Acc): 92.24. Result ID: 698e90ecffc3901de09eb86c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6b7f1e209131cea2e9dc.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6b7f1e209131cea2e9dc.md>)

[Implementation](<https://github.com/NervanaSystems/nlp-architect>)

Recorded evaluation configuration: {"Architecture":"BERT-Base","Precision":"8bit","Quantization Method":"Quantization-Aware Training (QAT)"}. This record does not assert independent reproduction.

### PERT

SST-2 (Acc): 92. Result ID: 698eb866ffc3901de0a01400.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Implementation](<https://github.com/ymcui/PERT>)

Recorded evaluation configuration: {"Scale":"base"}. This record does not assert independent reproduction.

### DQ BERT

SST-2 (Acc): 91.04. Result ID: 698e90ecffc3901de09eb86e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6b7f1e209131cea2e9dc.md>)

Recorded evaluation configuration: {"Architecture":"BERT-Base","Precision":"8bit","Quantization Method":"Dynamic Quantization (DQ)"}. This record does not assert independent reproduction.

### ALBERT

SST-2 (Acc): 90.6. Result ID: 698eb866ffc3901de0a01405.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa67061e209131cea13e88.md>)

[Implementation](<https://github.com/google-research/ALBERT>)

Recorded evaluation configuration: {"Scale":"large"}. This record does not assert independent reproduction.

### ALBERT

SST-2 (Acc): 89.4. Result ID: 698eb866ffc3901de0a013ff.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6c001e209131cea3193a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa67061e209131cea13e88.md>)

[Implementation](<https://github.com/google-research/ALBERT>)

Recorded evaluation configuration: {"Scale":"base"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
