Shot-Language Understanding on SLU OOD 11 unseen task variants (test)
55.8Composition Pattern ScoreAGENTSHOTS
Evaluation Results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AGENTSHOTSCategory=Ours2026.03 | 55.8 | 59.3 | 40 | 65 | 88 | 64.5 | 80 | 73 | 70.5 | 70 | 66.7 | 66.6 | 7 | |
| Gemini-3.0-ProCategory=General-purpose VLMs2026.03 | 52.5 | 43.6 | 43.3 | 79.2 | 58.5 | 44.1 | 35.5 | 52 | 64.5 | 69.1 | 60 | 54.8 | 0 | |
| UNISHOTCategory=Ours2026.03 | 43.3 | 41.4 | 36.7 | 60.8 | 88 | 63.4 | 80 | 72 | 67.5 | 62.7 | 63.3 | 61.7 | 2 | |
| Gemini-2.5-FlashCategory=General-purpose VLMs2026.03 | 20.8 | 48.6 | 33.3 | 57.5 | 75 | 45.2 | 55.5 | 61.5 | 68 | 59.1 | 60 | 53.1 | 0 | |
| Gemini-3.0-FlashCategory=General-purpose VLMs2026.03 | 15.8 | 60 | 46.7 | 82.5 | 81 | 57 | 57 | 56.5 | 60.5 | 60 | 61.7 | 58.1 | 3 | |
| ShotVL-7BCategory=Existing SLU2026.03 | 13.3 | 50.7 | 30 | 60.8 | 80.5 | 30.1 | 49 | 61.5 | 73 | 58.2 | 55 | 51.1 | 1 | |
| Qwen3-VL-8BCategory=General-purpose VLMs2026.03 | 8.3 | 37.1 | 21.7 | 65 | 81.5 | 36.6 | 53.5 | 64 | 51 | 58.2 | 48.3 | 47.8 | 0 |