# Specificity Testing on S 1.0 (test)

Source: [Specificity Testing on S 1.0 (test)](<https://www.sota2.com/research/sota/specificity-testing-on-s-1-0-test>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:37:49.624Z

Evaluation of model performance on Specificity Testing using the S 1.0 dataset (test split), measured by metrics such as birthplace, occupation, and location accuracy.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Specificity Testing |
| Dataset (including any stated split) | S 1.0 (test) |
| Metric | Birthplace Accuracy |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Birthplace Accuracy](<https://www.sota2.com/research/sota/specificity-testing-on-s-1-0-test.md?metric=Birthplace%20Accuracy>): Higher is better; unit: PERCENT.

- [Occupation Accuracy](<https://www.sota2.com/research/sota/specificity-testing-on-s-1-0-test.md?metric=Occupation%20Accuracy>): Higher is better; unit: PERCENT.

- [Location Accuracy](<https://www.sota2.com/research/sota/specificity-testing-on-s-1-0-test.md?metric=Location%20Accuracy>): Higher is better; unit: PERCENT.

- [Subclass-of Accuracy](<https://www.sota2.com/research/sota/specificity-testing-on-s-1-0-test.md?metric=Subclass-of%20Accuracy>): Higher is better; unit: PERCENT.

- [Part-of Accuracy](<https://www.sota2.com/research/sota/specificity-testing-on-s-1-0-test.md?metric=Part-of%20Accuracy>): Higher is better; unit: PERCENT.

- [Average Accuracy](<https://www.sota2.com/research/sota/specificity-testing-on-s-1-0-test.md?metric=Average%20Accuracy>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Birthplace Accuracy | Result date | Report paper |
| --- | --- | --- | --- |
| RoBERTa-Large (CP) | 89 | Not recorded | 68fa6f7e1e209131cea45f7e |
| BERT-Large (CP) | 82.25 | Not recorded | 68fa6f7e1e209131cea45f7e |
| GPT-2 (FP) | 81.01 | Not recorded | 68fa6f7e1e209131cea45f7e |
| RoBERTa-Large (FP) | 70.51 | Not recorded | 68fa6f7e1e209131cea45f7e |
| BERT-Base (FP) | 67.85 | Not recorded | 68fa6f7e1e209131cea45f7e |
| BERT-Large (FP) | 66.17 | Not recorded | 68fa6f7e1e209131cea45f7e |
| RoBERTa-Base (FP) | 64.85 | Not recorded | 68fa6f7e1e209131cea45f7e |
| RoBERTa-Base (CP) | 63.09 | Not recorded | 68fa6f7e1e209131cea45f7e |
| BERT-Base (VP) | 60.68 | Not recorded | 68fa6f7e1e209131cea45f7e |
| GPT-2 (VP) | 59.72 | Not recorded | 68fa6f7e1e209131cea45f7e |
| GPT-2 (CP)\* | 59.72 | Not recorded | 68fa6f7e1e209131cea45f7e |
| BERT-Base (CP) | 59.68 | Not recorded | 68fa6f7e1e209131cea45f7e |
| BERT-Large (VP) | 56.52 | Not recorded | 68fa6f7e1e209131cea45f7e |
| RoBERTa-Base (VP) | 54.48 | Not recorded | 68fa6f7e1e209131cea45f7e |
| RoBERTa-Large (VP) | 42.16 | Not recorded | 68fa6f7e1e209131cea45f7e |

### RoBERTa-Large (CP)

Birthplace Accuracy: 89. Result ID: 6990d9c73a6c31484903c522.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Prompting Technique":"Cascade Prompting"}. This record does not assert independent reproduction.

### BERT-Large (CP)

Birthplace Accuracy: 82.25. Result ID: 6990d9c73a6c31484903c51c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"BERT-Large","Prompting Technique":"Cascade Prompting"}. This record does not assert independent reproduction.

### GPT-2 (FP)

Birthplace Accuracy: 81.01. Result ID: 6990d9c73a6c31484903c515.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"GPT-2","Prompting Technique":"Few-shot Prompting","Number of demonstrations (K)":10}. This record does not assert independent reproduction.

### RoBERTa-Large (FP)

Birthplace Accuracy: 70.51. Result ID: 6990d9c73a6c31484903c521.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Prompting Technique":"Few-shot Prompting","Number of demonstrations (K)":10}. This record does not assert independent reproduction.

### BERT-Base (FP)

Birthplace Accuracy: 67.85. Result ID: 6990d9c73a6c31484903c518.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"BERT-Base","Prompting Technique":"Few-shot Prompting","Number of demonstrations (K)":10}. This record does not assert independent reproduction.

### BERT-Large (FP)

Birthplace Accuracy: 66.17. Result ID: 6990d9c73a6c31484903c51b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"BERT-Large","Prompting Technique":"Few-shot Prompting","Number of demonstrations (K)":10}. This record does not assert independent reproduction.

### RoBERTa-Base (FP)

Birthplace Accuracy: 64.85. Result ID: 6990d9c73a6c31484903c51e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"RoBERTa-Base","Prompting Technique":"Few-shot Prompting","Number of demonstrations (K)":10}. This record does not assert independent reproduction.

### RoBERTa-Base (CP)

Birthplace Accuracy: 63.09. Result ID: 6990d9c73a6c31484903c51f.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"RoBERTa-Base","Prompting Technique":"Cascade Prompting"}. This record does not assert independent reproduction.

### BERT-Base (VP)

Birthplace Accuracy: 60.68. Result ID: 6990d9c73a6c31484903c517.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65c31e209131cea0c4ed.md>)

[Implementation](<https://github.com/google-research/bert>)

Recorded evaluation configuration: {"Backbone":"BERT-Base","Prompting Technique":"Vanilla Prompting"}. This record does not assert independent reproduction.

### GPT-2 (VP)

Birthplace Accuracy: 59.72. Result ID: 6990d9c73a6c31484903c514.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

Recorded evaluation configuration: {"Backbone":"GPT-2","Prompting Technique":"Vanilla Prompting"}. This record does not assert independent reproduction.

### GPT-2 (CP)\*

Birthplace Accuracy: 59.72. Result ID: 6990d9c73a6c31484903c516.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"GPT-2","Prompting Technique":"Cascade Prompting","Rescore all suffixes":false}. This record does not assert independent reproduction.

### BERT-Base (CP)

Birthplace Accuracy: 59.68. Result ID: 6990d9c73a6c31484903c519.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Implementation](<https://github.com/jeffhj/S-TEST>)

Recorded evaluation configuration: {"Backbone":"BERT-Base","Prompting Technique":"Cascade Prompting"}. This record does not assert independent reproduction.

### BERT-Large (VP)

Birthplace Accuracy: 56.52. Result ID: 6990d9c73a6c31484903c51a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65c31e209131cea0c4ed.md>)

[Implementation](<https://github.com/google-research/bert>)

Recorded evaluation configuration: {"Backbone":"BERT-Large","Prompting Technique":"Vanilla Prompting"}. This record does not assert independent reproduction.

### RoBERTa-Base (VP)

Birthplace Accuracy: 54.48. Result ID: 6990d9c73a6c31484903c51d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66131e209131cea0e18b.md>)

[Implementation](<https://github.com/pytorch/fairseq>)

Recorded evaluation configuration: {"Backbone":"RoBERTa-Base","Prompting Technique":"Vanilla Prompting"}. This record does not assert independent reproduction.

### RoBERTa-Large (VP)

Birthplace Accuracy: 42.16. Result ID: 6990d9c73a6c31484903c520.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f7e1e209131cea45f7e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66131e209131cea0e18b.md>)

[Implementation](<https://github.com/pytorch/fairseq>)

Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Prompting Technique":"Vanilla Prompting"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
