# Code Prediction by Feeding Trees to Transformers

Source: [Code Prediction by Feeding Trees to Transformers](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468da>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:08:36.300Z

Authors: Seohyun Kim, Jinman Zhao, Yuchi Tian, Satish Chandra

Published: 2020-03-30T22:38:28.000Z

## Abstract

We advance the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems. First, we report that using the recently proposed Transformer architecture even out-of-the-box outperforms previous neural and non-neural systems for code prediction. We then show that by making the Transformer architecture aware of the syntactic structure of code, we further increase the margin by which a Transformer-based system outperforms previous systems. With this, it outperforms the accuracy of an RNN-based system (similar to Hellendoorn et al. 2018) by 18.3%, the Deep3 system (Raychev et al 2016) by 14.1%, and an adaptation of Code2Seq (Alon et al., 2018) for code prediction by 14.4%.   We present in the paper several ways of communicating the code structure to the Transformer, which is fundamentally built for processing sequence data. We provide a comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on a Facebook internal Python corpus. Our code and data preparation pipeline will be available in open source.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Next token prediction on py150](<https://www.sota2.com/research/sota/next-token-prediction-on-py150.md>): Next-token prediction; PY150; MRR: 67.2.

- [Next token prediction on internal](<https://www.sota2.com/research/sota/next-token-prediction-on-internal.md>): Next-token prediction; Internal; MRR: 0.615.

- [Next Token Value Prediction on internal dataset](<https://www.sota2.com/research/sota/next-token-value-prediction-on-internal-dataset.md>): Next Token Value Prediction; Internal dataset; MRR (Attribute access): 44.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2003.13848>)

[PDF](<https://arxiv.org/pdf/2003.13848.pdf>)

[Implementation](<https://github.com/facebookresearch/code-prediction-transformer>)
