# Next token prediction on internal

Source: [Next token prediction on internal](<https://www.sota2.com/research/sota/next-token-prediction-on-internal>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:50:22.733Z

Evaluation of language model performance on the internal dataset, measured by MRR and OOV Rate.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Next token prediction |
| Dataset (including any stated split) | internal |
| Metric | MRR |
| Unit | SCALAR |
| Direction | Higher is better |
| Results loaded | 10 of 10 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [MRR](<https://www.sota2.com/research/sota/next-token-prediction-on-internal.md?metric=MRR>): Higher is better; unit: SCALAR.

- [OOV Rate](<https://www.sota2.com/research/sota/next-token-prediction-on-internal.md?metric=OOV%20Rate>): Lower is better; unit: PERCENT.

## Recorded results

| Method | MRR | Result date | Report paper |
| --- | --- | --- | --- |
| TravTrans | 0.615 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 0.552 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 0.533 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 0.507 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 0.504 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 0.463 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 0.447 | Not recorded | 68fa6f9a1e209131cea468da |
| TravTrans | 0.439 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 0.421 | Not recorded | 68fa6f9a1e209131cea468da |
| PointerMixture | 0.404 | Not recorded | 68fa6f9a1e209131cea468da |

### TravTrans

MRR: 0.615. Result ID: 6990cf353a6c3148490372f7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Numeric constant"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 0.552. Result ID: 6990cf353a6c3148490372fa.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Function parameter name"}. This record does not assert independent reproduction.

### TravTrans

MRR: 0.533. Result ID: 6990cf353a6c3148490372fb.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Function parameter name"}. This record does not assert independent reproduction.

### TravTrans

MRR: 0.507. Result ID: 6990cf353a6c3148490372f9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Name (variable, module)"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 0.504. Result ID: 6990cf353a6c3148490372f4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Attribute access"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 0.463. Result ID: 6990cf353a6c3148490372fc.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"All leaf nodes"}. This record does not assert independent reproduction.

### TravTrans

MRR: 0.447. Result ID: 6990cf353a6c3148490372f5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"Attribute access"}. This record does not assert independent reproduction.

### TravTrans

MRR: 0.439. Result ID: 6990cf353a6c3148490372fd.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)

Recorded evaluation configuration: {"Application":"All leaf nodes"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 0.421. Result ID: 6990cf353a6c3148490372f8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Name (variable, module)"}. This record does not assert independent reproduction.

### PointerMixture

MRR: 0.404. Result ID: 6990cf353a6c3148490372f6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66551e209131cea0f8ae.md>)

Recorded evaluation configuration: {"Application":"Numeric constant"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
