Return type hint prediction on TypeWriter OSS 25% subset
78.8PrecisionTypeWriter
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TypeWriterSupervision=Supervised2022.04 | 78.8 | 69.9 | 74.1 | |
| INCODER-6.7BInference=Infill, Return checks=true2022.04 | 76.7 | 76.7 | 76.7 | |
| INCODER-6.7BInference=Left-to-right rerank, Return checks=true2022.04 | 64.3 | 64.3 | 64.3 | |
| INCODER-6.7BInference=Left-to-right single, Return checks=true2022.04 | 63.2 | 63.2 | 63.2 | |
| INCODER-6.7BInference=Infill, Return checks=false2022.04 | 46.8 | 46.8 | 46.8 | |
| INCODER-6.7BInference=Left-to-right rerank, Return checks=false2022.04 | 24.2 | 24.2 | 24.2 | |
| INCODER-6.7BInference=Left-to-right single, Return checks=false2022.04 | 20 | 20 | 20 |