Medical Visual Question Answering on Combined SLAKE/VQA-Rad/VQA-Med (Test, Mixed-Modality Task 3)
82.33SLAKE ScoreOmniFM
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| OmniFMTuning Protocol=Full Tuning, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 82.33 | 70.21 | 18.24 | 22.35 | 43.37 | 47.3 | |
| FedPerTuning Protocol=Full Tuning, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 79.23 | 69.2 | 16.77 | 22.54 | 43.43 | 46.03 | |
| FedDynTuning Protocol=Full Tuning, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 78.24 | 69 | 16.21 | 21.65 | 42.02 | 45.02 | |
| OmniFMTuning Protocol=Adapter, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 77.85 | 67.42 | 13.79 | 21.89 | 40.12 | 44.21 | |
| OmniFM (Ours)Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 77.85 | 67.42 | 13.79 | 21.89 | 40.12 | 44.21 | |
| FedProxTuning Protocol=Full Tuning, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 77.47 | 67 | 14.87 | 22.12 | 41 | 44.33 | |
| FedAvgTuning Protocol=Full Tuning, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 77.45 | 67.25 | 15.06 | 22 | 41.52 | 44.66 | |
| FedAdamTuning Protocol=Full Tuning, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 77 | 67.89 | 15.12 | 22.44 | 42.42 | 44.77 | |
| FedPerTuning Protocol=Adapter, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 75.1 | 66.3 | 12.95 | 21.76 | 39.7 | 43.16 | |
| F3OCUSVisual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 74.69 | 60.05 | 12.84 | 24 | 42.3 | 42.78 | |
| FedDynTuning Protocol=Adapter, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 74.02 | 65.88 | 12.41 | 21.39 | 39.05 | 42.55 | |
| FedProxTuning Protocol=Adapter, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 73.1 | 64.7 | 11.72 | 21.18 | 38.6 | 41.86 | |
| OmniFMTuning Protocol=LayerNorm, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 73.1 | 64.6 | 12.35 | 22.61 | 38.52 | 42.24 | |
| FedAdamTuning Protocol=Adapter, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 73.05 | 65.12 | 12.08 | 21.47 | 39.12 | 42.17 | |
| FedAvgTuning Protocol=Adapter, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 72.82 | 64.45 | 11.56 | 21 | 38.35 | 41.64 | |
| FedDATVisual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 72.79 | 63.58 | 12.46 | 23 | 38.91 | 42.15 | |
| OmniFMTuning Protocol=Bias, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 71.25 | 57.1 | 12.35 | 17.61 | 36.22 | 38.91 | |
| FedPerTuning Protocol=LayerNorm, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 71.22 | 63.88 | 11.7 | 22.55 | 38.1 | 41.49 | |
| FedDynTuning Protocol=LayerNorm, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 70.55 | 63.4 | 11.32 | 22.28 | 37.6 | 41.03 | |
| FedPerTuning Protocol=Bias, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 70.1 | 56.88 | 12 | 17 | 36 | 38.4 | |
| FedAdamTuning Protocol=LayerNorm, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 70.02 | 62.95 | 11.08 | 22.35 | 37.48 | 40.78 | |
| FedProxTuning Protocol=LayerNorm, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 69.8 | 62.4 | 10.75 | 22.12 | 37.05 | 40.42 | |
| FedAvgTuning Protocol=LayerNorm, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 69.53 | 62.22 | 10.59 | 22 | 36.89 | 40.24 | |
| FedDynTuning Protocol=Bias, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 68.7 | 56 | 11.45 | 17.05 | 35.7 | 37.78 | |
| FedProxTuning Protocol=Bias, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 68.6 | 55.4 | 11.35 | 17.12 | 35.6 | 37.61 | |
| FedAvgTuning Protocol=Bias, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 68.25 | 55.61 | 11.17 | 17 | 35.47 | 37.5 | |
| FedAdamTuning Protocol=Bias, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 68.1 | 56.05 | 11.7 | 17.35 | 36.1 | 37.86 | |
| PromptFLVisual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 65.63 | 57.56 | 4.78 | 15 | 40.26 | 36.65 | |
| OmniFMTuning Protocol=LoRA, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 62.45 | 62.1 | 6.4 | 15.66 | 32.05 | 35.73 | |
| FedPerTuning Protocol=LoRA, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 60.2 | 61.55 | 5.8 | 15.7 | 32.4 | 35.13 | |
| FedDynTuning Protocol=LoRA, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 58.3 | 60.7 | 5.05 | 15.1 | 31.6 | 34.15 | |
| FedProxTuning Protocol=LoRA, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 58.1 | 60.05 | 4.8 | 15.21 | 31.45 | 33.92 | |
| FedAdamTuning Protocol=LoRA, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 58.02 | 60.72 | 5.1 | 15.48 | 31.9 | 34.24 | |
| FedAvgTuning Protocol=LoRA, Visual Encoder=CLIP ViT-B/32, Text Backbone=LLaMA-3.2-3B2026.03 | 57.73 | 60.18 | 4.59 | 15 | 31.16 | 33.73 |