Grammaticality Judgment on RuBLiMP Russian 1.0
96.7AnaphoraHaiku (GPb+CT son)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Haiku (GPb+CT son)Backbone=Haiku, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=son2025.06 | 96.7 | 74 | 100 | 94 | 100 | 97.3 | 100 | 94.6 | |
| gpt-4o (CT)Backbone=gpt-4o, Prompting=Chain-of-Thought (CT)2025.06 | 96.7 | 87.3 | 96.7 | 95.9 | 86.7 | 96 | 100 | 94.2 | |
| gpt-4o (GPb+CT son)Backbone=gpt-4o, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=son2025.06 | 96.7 | 88 | 100 | 98.1 | 100 | 99 | 100 | 97.4 | |
| gpt-4o (GPb+CT ol)Backbone=gpt-4o, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=ol2025.06 | 96.7 | 84 | 100 | 95.1 | 99.8 | 98 | 99.3 | 96.1 | |
| Sonnet (base)Backbone=Sonnet, Prompting=base2025.06 | 96 | 90 | 96 | 96.8 | 96.2 | 97.7 | 98.7 | 95.9 | |
| Haiku (GPb+CT ol)Backbone=Haiku, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=ol2025.06 | 95.3 | 80.7 | 99.3 | 94.8 | 99.3 | 94 | 98 | 94.5 | |
| gpt-4o (GPb ol)Backbone=gpt-4o, Prompting=Grammar Prompting (GPb), Prompt Variant=ol2025.06 | 95.3 | 90 | 100 | 96.5 | 98.9 | 97 | 100 | 96.8 | |
| Sonnet (GPb son)Backbone=Sonnet, Prompting=Grammar Prompting (GPb), Prompt Variant=son2025.06 | 95.3 | 89.3 | 100 | 96.4 | 100 | 99.7 | 99.3 | 97.2 | |
| Llama (GPb+CT)Prompting Strategy=Grammar Prompting + Chain-of-Thought, Backbone=o12025.06 | 95.3 | 90.3 | 100 | 92 | 95.1 | 94.9 | 87.8 | 93.6 | |
| gpt-4o (base)Backbone=gpt-4o, Prompting=base2025.06 | 94.7 | 88.7 | 94.7 | 94.1 | 90.7 | 96.3 | 98.7 | 94 | |
| Sonnet (GPb ol)Backbone=Sonnet, Prompting=Grammar Prompting (GPb), Prompt Variant=ol2025.06 | 94.7 | 88 | 100 | 96.9 | 99.3 | 99 | 98.7 | 96.7 | |
| Sonnet (GPb+CT son)Backbone=Sonnet, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=son2025.06 | 94.7 | 89.3 | 100 | 97.7 | 100 | 99 | 100 | 97.2 | |
| Sonnet (GPb+CT ol)Backbone=Sonnet, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=ol2025.06 | 94.7 | 86.7 | 100 | 98.1 | 99.3 | 98.7 | 97.3 | 96.4 | |
| Sonnet (CT)Backbone=Sonnet, Prompting=Chain-of-Thought (CT)2025.06 | 94 | 89.3 | 93.3 | 97.2 | 92.2 | 97.3 | 99.3 | 94.7 | |
| gpt-3.5 (GPb+CT ol)Backbone=gpt-3.5, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=ol2025.06 | 93.3 | 70 | 100 | 89.5 | 97.3 | 82.7 | 79.3 | 87.4 | |
| gpt-4o (GPb son)Backbone=gpt-4o, Prompting=Grammar Prompting (GPb), Prompt Variant=son2025.06 | 93.3 | 90.7 | 100 | 95.6 | 100 | 98.3 | 100 | 96.8 | |
| Llama (CT)Prompting Strategy=Chain-of-Thought (CT)2025.06 | 93.3 | 90.5 | 96 | 94 | 79.9 | 86 | 95.3 | 90.7 | |
| gpt-3.5 (GPb+CT son)Backbone=gpt-3.5, Prompting=Grammar Prompting + CoT (GPb+CT), Prompt Variant=son2025.06 | 92.7 | 56.7 | 98.7 | 92.3 | 97.8 | 83.3 | 94.7 | 88 | |
| Haiku (GPb son)Backbone=Haiku, Prompting=Grammar Prompting (GPb), Prompt Variant=son2025.06 | 92.7 | 73.3 | 99.3 | 88.5 | 97.6 | 96.3 | 100 | 92.5 | |
| Haiku (GPb ol)Backbone=Haiku, Prompting=Grammar Prompting (GPb), Prompt Variant=ol2025.06 | 92.7 | 79.3 | 100 | 88.8 | 88 | 88.7 | 87.3 | 89.3 | |
| Llama (Base)Prompting Strategy=Basic2025.06 | 92.7 | 86.7 | 88 | 91.9 | 75.6 | 83.7 | 92 | 87.2 | |
| gpt-3.5 (base)Backbone=gpt-3.5, Prompting=base2025.06 | 92 | 68.7 | 84.7 | 84.1 | 58.9 | 74.3 | 90 | 79 | |
| Haiku (base)Backbone=Haiku, Prompting=base2025.06 | 91.3 | 84.7 | 90 | 90.1 | 63.6 | 88.3 | 98.7 | 86.7 | |
| Haiku (CT)Backbone=Haiku, Prompting=Chain-of-Thought (CT)2025.06 | 90.7 | 84.7 | 90.7 | 94 | 73.8 | 91 | 100 | 89.3 | |
| gpt-3.5 (GPb ol)Backbone=gpt-3.5, Prompting=Grammar Prompting (GPb), Prompt Variant=ol2025.06 | 89.3 | 77.3 | 100 | 84.5 | 86.4 | 69 | 94.7 | 85.9 | |
| gpt-3.5 (CT)Backbone=gpt-3.5, Prompting=Chain-of-Thought (CT)2025.06 | 88.7 | 77.3 | 68 | 87.3 | 64.7 | 69.3 | 90 | 77.9 | |
| gpt-3.5 (GPb son)Backbone=gpt-3.5, Prompting=Grammar Prompting (GPb), Prompt Variant=son2025.06 | 88.7 | 62 | 100 | 84.8 | 85.8 | 78 | 98.7 | 85.4 | |
| Llama (GPb)Prompting Strategy=Grammar Prompting (GPb), Backbone=o12025.06 | 87.3 | 87.3 | 100 | 89.5 | 91.1 | 88 | 87.3 | 90.1 |