Graph Reasoning on GVLQA BASE baseline (test)
99.35ConnectVicuna-13B
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Vicuna-13Bsetting=Fine-tuning, parameters=13B2024.02 | 99.35 | 94.39 | 36.73 | 28.53 | 11.34 | 92.65 | 34.81 | 56.83 | |
| GITA-13Bsetting=Fine-tuning, parameters=13B, vision-only=false2024.02 | 99.14 | 95.6 | 38.69 | 40.47 | 20.66 | 92.12 | 33.33 | 60 | |
| GITA-7Bsetting=Fine-tuning, parameters=7B, vision-only=false2024.02 | 98.95 | 96.67 | 41.12 | 32.15 | 20 | 93.19 | 29.26 | 58.76 | |
| LLaMA2-13Bsetting=Fine-tuning, parameters=13B2024.02 | 98.79 | 93.36 | 33.83 | 27.93 | 12.22 | 91.34 | 33.46 | 55.85 | |
| Vicuna-7Bsetting=Fine-tuning, parameters=7B2024.02 | 97.58 | 95.04 | 34.46 | 25.98 | 9.33 | 91.04 | 25.55 | 54.15 | |
| LLaMA2-7Bsetting=Fine-tuning, parameters=7B2024.02 | 97.33 | 94.63 | 33.26 | 26.01 | 9.56 | 90.86 | 23.95 | 53.66 | |
| GITA-ZSsetting=Zero-shot, vision-only=false2024.02 | 82.58 | 51.46 | 19.71 | 37.69 | 6 | 52.21 | 50 | 42.81 | |
| GPT-4 Turbosetting=Zero-shot2024.02 | 76.7 | 49.51 | 19.59 | 35.35 | 6.89 | 42.11 | 47.04 | 39.6 | |
| GITA-7B (VO)setting=Fine-tuning, parameters=7B, vision-only=true2024.02 | 59.97 | 96.34 | 13.3 | 5.72 | 2.89 | 93.01 | 1.11 | 38.91 | |
| GITA-13B (VO)setting=Fine-tuning, parameters=13B, vision-only=true2024.02 | 58 | 96.91 | 14.45 | 5.72 | 4.89 | 93.19 | 1.85 | 39.29 | |
| GITA-ZS (VO)setting=Zero-shot, vision-only=true2024.02 | 57.76 | 63.34 | 5.34 | 4.88 | 1.59 | 46.6 | 10.74 | 27.18 | |
| LLaMA2-7Bsetting=Zero-shot, parameters=7B2024.02 | 50.06 | 49.43 | 0 | 0 | 0 | 0 | 0 | 14.21 | |
| Vicuna-7Bsetting=Zero-shot, parameters=7B2024.02 | 50.06 | 49.43 | 0 | 0 | 0 | 0 | 0 | 14.21 | |
| GITA-7B (VO)setting=Zero-shot, parameters=7B, vision-only=true2024.02 | 50.06 | 50.33 | 0 | 0 | 0 | 0 | 0 | 14.34 | |
| GITA-7Bsetting=Zero-shot, parameters=7B, vision-only=false2024.02 | 50.06 | 49.43 | 0 | 0 | 0 | 0 | 0 | 14.21 |