Code Completion on Python Code Completion downstream
57.7Exact Similarity (ES)Gemini-3-Pro
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemini-3-ProInput Modality=Image2026.02 | 57.7 | 29.2 | |
| Gemini-3-FlashInput Modality=Image2026.02 | 57.1 | 29.2 | |
| Gemini-3-ProInput Modality=Text2026.02 | 55.8 | 27.6 | |
| Gemini-3-FlashInput Modality=Text2026.02 | 55.1 | 26.5 | |
| Gemini-2.5-ProInput Modality=Text2026.02 | 54.9 | 28.8 | |
| Gemini-2.5-ProInput Modality=Image2026.02 | 54.5 | 25.7 | |
| GPT-5-miniInput Modality=Image2026.02 | 51.6 | 24.7 | |
| GPT-5.1Input Modality=Text2026.02 | 51.6 | 24.6 | |
| GPT-5-miniInput Modality=Text2026.02 | 51.3 | 25.3 | |
| GLM-4.6vInput Modality=Image2026.02 | 50.8 | 17.2 | |
| Gemini-3-ProInput Modality=NoCtx2026.02 | 50.3 | 16.2 | |
| GLM-4.6vInput Modality=Text2026.02 | 49.8 | 21 | |
| Qwen-3-VLInput Modality=Text2026.02 | 49.7 | 21.6 | |
| Gemini-3-FlashInput Modality=NoCtx2026.02 | 49.7 | 15.8 | |
| GPT-5.1Input Modality=Image2026.02 | 47.9 | 18.6 | |
| Gemini-2.5-ProInput Modality=NoCtx2026.02 | 46.7 | 15.5 | |
| Qwen-3-VLInput Modality=NoCtx2026.02 | 45 | 12.8 | |
| GPT-5-miniInput Modality=NoCtx2026.02 | 42.6 | 10.7 | |
| GPT-5.1Input Modality=NoCtx2026.02 | 41.8 | 10.7 | |
| GLM-4.6vInput Modality=NoCtx2026.02 | 39.9 | 9.5 | |
| Qwen-3-VLInput Modality=Image2026.02 | 35.5 | 8 |