Compositional Reasoning on SugarCrepe 1.0 (test)
100Replace Acc (Object)Human
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Human2024.12 | 100 | 99 | 97 | 99 | 100 | 99 | 99 | |
| LAIONbackbone=xlm-roberta-large-ViT-H-142024.12 | 97 | 86 | 72 | 64 | 72 | 93 | 86 | |
| Ours (Grounded Recaptioning)training_data=64M synthetic hard negatives2024.12 | 97 | 94 | 88 | 89 | 94 | 95 | 93 | |
| DC-XL2024.12 | 96 | 85 | 70 | 65 | 67 | 91 | 85 | |
| GPT-4Vsees both captions simultaneously=true, two tower embedding model=false2024.12 | 96 | 94 | 90 | 83 | 90 | 92 | 92 | |
| CLIP2024.12 | 94 | 79 | 65 | 60 | 62 | 78 | 72 | |
| CapPaobjective=captioning2024.12 | 92 | 90 | 87 | 82 | 88 | 99 | 99 | |
| Veratext-only=true, sees images=false2024.12 | 49 | 50 | 49 | 49 | 49 | 49 | 50 |