Intent Grasping on IntentGrasp Gem Set
0.2499F1 ScoreGemini 3-Flash
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Gemini 3-Flashshuffle=no shuffle2026.05 | 0.2499 | — | — | |
| Gemini 3-Flashshuffle=shuffle-72026.05 | 0.2498 | — | — | |
| Gemini 3-Flashshuffle=shuffle-422026.05 | 0.2495 | — | — | |
| Gemini 3-Flashshuffle=shuffle-3652026.05 | 0.2375 | — | — | |
| Gemini 3.1-Proshuffle=shuffle-72026.05 | 0.2214 | — | — | |
| Gemini 3.1-Proshuffle=shuffle-422026.05 | 0.2184 | — | — | |
| Gemini 3.1-Proshuffle=no shuffle2026.05 | 0.2154 | — | — | |
| Gemini 3.1-Proshuffle=shuffle-3652026.05 | 0.2051 | — | — | |
| Claude Opus-4.7shuffle=shuffle-422026.05 | 0.1801 | — | — | |
| Claude Opus-4.7shuffle=no shuffle2026.05 | 0.1623 | — | — | |
| Claude Opus-4.7shuffle=shuffle-72026.05 | 0.1614 | — | — | |
| Claude Opus-4.7shuffle=shuffle-3652026.05 | 0.1581 | — | — | |
| GPT 5.4shuffle=shuffle-422026.05 | 0.1275 | — | — | |
| Claude Sonnet-4.6shuffle=shuffle-422026.05 | 0.1238 | — | — | |
| Gemini 3.1-Flash-Liteshuffle=shuffle-422026.05 | 0.1212 | — | — | |
| GPT 5.4shuffle=shuffle-3652026.05 | 0.1189 | — | — | |
| Claude Sonnet-4.6shuffle=shuffle-72026.05 | 0.1184 | — | — | |
| Claude Sonnet-4.6shuffle=shuffle-3652026.05 | 0.1174 | — | — | |
| Gemini 3.1-Flash-Liteshuffle=shuffle-3652026.05 | 0.1167 | — | — | |
| Gemini 3.1-Flash-Liteshuffle=no shuffle2026.05 | 0.1166 | — | — | |
| Gemini 3.1-Flash-Liteshuffle=shuffle-72026.05 | 0.1162 | — | — | |
| GPT 5.4shuffle=shuffle-72026.05 | 0.114 | — | — | |
| GPT 5.4shuffle=no shuffle2026.05 | 0.1079 | — | — | |
| GPT 5.4-nanoshuffle=shuffle-422026.05 | 0.0898 | — | — | |
| GPT 5.4-nanoshuffle=no shuffle2026.05 | 0.0863 | — | — | |
| GPT 5.4-nanoshuffle=shuffle-72026.05 | 0.0862 | — | — | |
| GPT 5.4-minishuffle=shuffle-3652026.05 | 0.0855 | — | — | |
| GPT 5.4-nanoshuffle=shuffle-3652026.05 | 0.0847 | — | — | |
| Claude Sonnet-4.6shuffle=no shuffle2026.05 | 0.0766 | — | — | |
| Claude Haiku-4.5shuffle=shuffle-422026.05 | 0.0723 | — | — | |
| GPT 5.4-minishuffle=shuffle-422026.05 | 0.0688 | — | — | |
| GPT 5.4-minishuffle=shuffle-72026.05 | 0.067 | — | — | |
| GPT 5.4-minishuffle=no shuffle2026.05 | 0.0617 | — | — | |
| Claude Haiku-4.5shuffle=shuffle-3652026.05 | 0.0585 | — | — | |
| Claude Haiku-4.5shuffle=shuffle-72026.05 | 0.054 | — | — | |
| Claude Haiku-4.5shuffle=no shuffle2026.05 | 0.0454 | — | — | |
| Claude Haiku-4.5aggregation=Average across shuffles2026.05 | — | 0.0576 | 0.0097 | |
| Claude Opus-4.7aggregation=Average across shuffles2026.05 | — | 0.1655 | 0.0086 | |
| Claude Sonnet-4.6aggregation=Average across shuffles2026.05 | — | 0.1091 | 0.0189 | |
| Gemini 3-Flashaggregation=Average across shuffles2026.05 | — | 0.2467 | 0.0053 | |
| Gemini 3.1-Flash-Liteaggregation=Average across shuffles2026.05 | — | 0.1177 | 0.002 | |
| Gemini 3.1-Proaggregation=Average across shuffles2026.05 | — | 0.2151 | 0.0061 | |
| GPT 5.4aggregation=Average across shuffles2026.05 | — | 0.1171 | 0.0071 | |
| GPT 5.4-miniaggregation=Average across shuffles2026.05 | — | 0.0708 | 0.0089 | |
| GPT 5.4-nanoaggregation=Average across shuffles2026.05 | — | 0.0868 | 0.0019 |