Multi-image Reasoning on DEMON
38.94AccuracyBrote-IM-XXL
Evaluation results
| Method | Links | |
|---|---|---|
| Brote-IM-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 38.94 | |
| Brote-IM-XXL (ours)#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 38.94 | |
| Brote-EX-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 38.7 | |
| Brote-EX-XXL (ours)#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 38.7 | |
| MMICL-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 38.14 | |
| MMICL-XL#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 38.14 | |
| Brote-IM-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 37.34 | |
| Brote-IM-XL (ours)#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 37.34 | |
| Brote-EX-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 37.33 | |
| Brote-EX-XL (ours)#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 37.33 | |
| VPG-C-LLAMA2LLM Backbone=LLAMA, #Param LLM=7B2024.02 | 37.22 | |
| VPG-C-LLaMA2#Param LLM=7B, Task Setting=Multi-image / Video Tasks2024.02 | 37.22 | |
| VPG-C-VicunaLLM Backbone=Vicuna, #Param LLM=13B2024.02 | 36.37 | |
| MMICL-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 36.3 | |
| MMICL-XXL#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 36.3 | |
| InstructBLIP-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 33 | |
| InstructBLIP-XXL#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 33 | |
| InstructBLIP-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 32.59 | |
| InstructBLIP-XL#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 32.59 | |
| OtterLLM Backbone=MPT, #Param LLM=7B2024.02 | 24.51 | |
| Otter#Param LLM=7B, Task Setting=Multi-image / Video Tasks2024.02 | 24.51 |