# Next Token Value Prediction on internal dataset

Source: [Next Token Value Prediction on internal dataset](<https://www.sota2.com/research/sota/next-token-value-prediction-on-internal-dataset>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:50:31.281Z

Evaluation of model performance on the internal dataset using Mean Reciprocal Rank (MRR) across specific structural components (e.g., attribute access, variable names, leaf nodes).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Next Token Value Prediction |
| Dataset (including any stated split) | internal dataset |
| Metric | MRR (Attribute access) |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 4 of 4 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [MRR (Attribute access)](<https://www.sota2.com/research/sota/next-token-value-prediction-on-internal-dataset.md?metric=MRR%20(Attribute%20access)>): Higher is better; unit: PERCENT.

- [MRR (Numeric constant)](<https://www.sota2.com/research/sota/next-token-value-prediction-on-internal-dataset.md?metric=MRR%20(Numeric%20constant)>): Higher is better; unit: PERCENT.

- [MRR (Name)](<https://www.sota2.com/research/sota/next-token-value-prediction-on-internal-dataset.md?metric=MRR%20(Name)>): Higher is better; unit: PERCENT.

- [MRR (Function Parameter Name)](<https://www.sota2.com/research/sota/next-token-value-prediction-on-internal-dataset.md?metric=MRR%20(Function%20Parameter%20Name)>): Higher is better; unit: PERCENT.

- [MRR (All Leaf Nodes)](<https://www.sota2.com/research/sota/next-token-value-prediction-on-internal-dataset.md?metric=MRR%20(All%20Leaf%20Nodes)>): Higher is better; unit: PERCENT.

## Recorded results

| Method | MRR (Attribute access) | Result date | Report paper |
| --- | --- | --- | --- |
| TravTrans | 44.7 | Not recorded | 68fa6f9a1e209131cea468da |
| PathTrans | 41.5 | Not recorded | 68fa6f9a1e209131cea468da |
| Deep3 | 38.5 | Not recorded | 68fa6f9a1e209131cea468da |
| Code2Seq | 26.4 | Not recorded | 68fa6f9a1e209131cea468da |

### TravTrans

MRR (Attribute access): 44.7. Result ID: 6990cf353a6c314849037301.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### PathTrans

MRR (Attribute access): 41.5. Result ID: 6990cf353a6c314849037300.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### Deep3

MRR (Attribute access): 38.5. Result ID: 6990cf353a6c3148490372fe.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### Code2Seq

MRR (Attribute access): 26.4. Result ID: 6990cf353a6c3148490372ff.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65641e209131cea09eb4.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
