# Next token prediction on py150

Source: [Next token prediction on py150](<https://www.sota2.com/research/sota/next-token-prediction-on-py150>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:50:22.718Z

Evaluation of language model performance on the py150 dataset, measured using Mean Reciprocal Rank (MRR).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Next token prediction |
| Dataset (including any stated split) | py150 |
| Metric | MRR |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 20 of 20 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [MRR](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md?metric=MRR>): Higher is better; unit: PERCENT.

- [OOV Rate](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md?metric=OOV%20Rate>): Lower is better; unit: PERCENT.

- [MRR (Attribute access)](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md?metric=MRR%20(Attribute%20access)>): Higher is better; unit: PERCENT.

- [MRR (Numeric constant)](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md?metric=MRR%20(Numeric%20constant)>): Higher is better; unit: PERCENT.

- [MRR (Name (variable, module))](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md?metric=MRR%20(Name%20(variable%2C%20module))>): Higher is better; unit: PERCENT.

- [MRR (Function parameter name)](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md?metric=MRR%20(Function%20parameter%20name)>): Higher is better; unit: PERCENT.

- [MRR (All leaf tokens)](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md?metric=MRR%20(All%20leaf%20tokens)>): Higher is better; unit: PERCENT.

## Recorded results

| Method | MRR | Result date | Report paper |
| --- | --- | --- | --- |
| TravTrans | 67.2 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 66.6 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 63.5 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 60.5 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 59.7 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 58 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 58 | Not recorded | 68fa6f9a1e209131cea468da |
| PathTrans | 55.1 | Not recorded | 68fa6f9a1e209131cea468da |
| SeqTrans | 54.9 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 54.2 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 51.9 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 50.6 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 49.5 | Not recorded | 68fa6f9a1e209131cea468da |
| Deep3 | 43.9 | Not recorded | 68fa6f9a1e209131cea468da |
| Code2Seq | 43.7 | Not recorded | 68fa6f9a1e209131cea468da |
| SeqRNN | 36.6 | Not recorded | 68fa6f9a1e209131cea468da |

### TravTrans

MRR: 67.2. Result ID: 6990cf353a6c3148490372f1.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Function parameter name"}. This record does not assert independent reproduction.

### TravTrans

MRR: 66.6. Result ID: 6990cf353a6c3148490372ef.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Name (variable, module)"}. This record does not assert independent reproduction.

### TravTrans

MRR: 63.5. Result ID: 6990cf353a6c3148490372ed.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Numeric constant"}. This record does not assert independent reproduction.

### TravTrans

MRR: 60.5. Result ID: 6990cf353a6c3148490372eb.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Attribute access"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 59.7. Result ID: 6990cf353a6c3148490372f0.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Function parameter name"}. This record does not assert independent reproduction.

### TravTrans

MRR: 58. Result ID: 6990cf353a6c3148490372e5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Architecture":"Attention","Input Representation":"AST"}. This record does not assert independent reproduction.

### TravTrans

MRR: 58. Result ID: 6990cf353a6c3148490372f3.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"All leaf nodes"}. This record does not assert independent reproduction.

### PathTrans

MRR: 55.1. Result ID: 6990cf353a6c3148490372e6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Architecture":"Attention","Input Representation":"AST"}. This record does not assert independent reproduction.

### SeqTrans

MRR: 54.9. Result ID: 6990cf353a6c3148490372e4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Architecture":"Transformer","Input Representation":"Sequence"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 54.2. Result ID: 6990cf353a6c3148490372ea.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Attribute access"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 51.9. Result ID: 6990cf353a6c3148490372f2.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"All leaf nodes"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 50.6. Result ID: 6990cf353a6c3148490372ec.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Numeric constant"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 49.5. Result ID: 6990cf353a6c3148490372ee.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Name (variable, module)"}. This record does not assert independent reproduction.

### Deep3

MRR: 43.9. Result ID: 6990cf353a6c3148490372e9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

Recorded evaluation configuration: {"Architecture":"Decision tree","Input Representation":"AST"}. This record does not assert independent reproduction.

### Code2Seq

MRR: 43.7. Result ID: 6990cf353a6c3148490372e7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65641e209131cea09eb4.md>)

Recorded evaluation configuration: {"Architecture":"Attention","Input Representation":"AST"}. This record does not assert independent reproduction.

### SeqRNN

MRR: 36.6. Result ID: 6990cf353a6c3148490372e8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65851e209131cea0acbc.md>)

Recorded evaluation configuration: {"Architecture":"RNN","Input Representation":"Sequence"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
