Python function return type prediction on TypeWriter OSS non-None return types (test)
59.2PrecisionInCoder (Causal-masked infilling)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| InCoder (Causal-masked infilling)Inference strategy=Causal-masked infilling, Learning paradigm=Zero-shot, Generation=Greedy2022.04 | 59.2 | 59.2 | 59.2 | |
| TypeWriterLearning paradigm=Supervised2022.04 | 54.9 | 43.2 | 48.3 | |
| InCoder (Left-to-right reranking)Inference strategy=Left-to-right reranking, Learning paradigm=Zero-shot, Temperature=0.82022.04 | 33.3 | 33.3 | 33.3 | |
| InCoder (Left-to-right single)Inference strategy=Left-to-right single, Learning paradigm=Zero-shot, Temperature=0.22022.04 | 30.8 | 30.8 | 30.8 |