# Multimodal Autocomplete on ImageChat (unseen prefixes)

Source: [Multimodal Autocomplete on ImageChat (unseen prefixes)](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:55:16.411Z

Evaluation of multimodal autocomplete performance on the unseen prefixes subset of the ImageChat dataset, measured using metrics like PR-F1, Semantic Match (SM), and inference time.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Multimodal Autocomplete |
| Dataset (including any stated split) | ImageChat (unseen prefixes) |
| Metric | PR-F1 |
| Unit | SCALAR |
| Direction | Higher is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [PR-F1](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=PR-F1>): Higher is better; unit: SCALAR.

- [SM](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=SM>): Higher is better; unit: SCALAR.

- [Time (s)](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=Time%20(s)>): Lower is better; unit: TIME.

- [TR](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=TR>): Higher is better; unit: SCALAR.

- [PR-P](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=PR-P>): Higher is better; unit: SCALAR.

- [PR-R](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=PR-R>): Higher is better; unit: SCALAR.

- [Prediction Length](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=Prediction%20Length>): Higher is better; unit: SCALAR.

- [TES (s)](<https://www.sota2.com/research/sota/multimodal-autocomplete-on-imagechat-unseen-prefixes.md?metric=TES%20(s)>): Lower is better; unit: TIME.

## Recorded results

| Method | PR-F1 | Result date | Report paper |
| --- | --- | --- | --- |
| Router-4-Max (Oracle) | 0.281 | Not recorded | 69647fda53fdd5c3617445a9 |
| MiniCPM-V | 0.2586 | Not recorded | 69647fda53fdd5c3617445a9 |
| Qwen2-VL | 0.2356 | Not recorded | 69647fda53fdd5c3617445a9 |
| PaliGemma | 0.2348 | Not recorded | 69647fda53fdd5c3617445a9 |
| MiniCPM-V | 0.223 | Not recorded | 69647fda53fdd5c3617445a9 |
| Router-4-L | 0.212 | Not recorded | 69647fda53fdd5c3617445a9 |
| Router-4-P | 0.212 | Not recorded | 69647fda53fdd5c3617445a9 |
| PaliGemma | 0.199 | Not recorded | 69647fda53fdd5c3617445a9 |
| Qwen2-VL | 0.197 | Not recorded | 69647fda53fdd5c3617445a9 |
| Router-2-L | 0.196 | Not recorded | 69647fda53fdd5c3617445a9 |
| Router-2-P | 0.196 | Not recorded | 69647fda53fdd5c3617445a9 |
| QB | 0.1755 | Not recorded | 69647fda53fdd5c3617445a9 |
| MPC++ | 0.1523 | Not recorded | 69647fda53fdd5c3617445a9 |
| QB | 0.135 | Not recorded | 69647fda53fdd5c3617445a9 |
| MPC | 0.0845 | Not recorded | 69647fda53fdd5c3617445a9 |

### Router-4-Max (Oracle)

PR-F1: 0.281. Result ID: 699693995698665ec4dcd9ff.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Implementation](<https://github.com/devichand579/MAC>)

Recorded evaluation configuration: {"Router configuration":"Router-4","Oracle":true}. This record does not assert independent reproduction.

### MiniCPM-V

PR-F1: 0.2586. Result ID: 699693995698665ec4dcda09.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74681e209131cea6183d.md>)

[Implementation](<https://github.com/OpenBMB/MiniCPM-V>)

Recorded evaluation configuration: {"Type":"VLMs"}. This record does not assert independent reproduction.

### Qwen2-VL

PR-F1: 0.2356. Result ID: 699693995698665ec4dcda0b.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75151e209131cea65976.md>)

[Implementation](<https://github.com/QwenLM/Qwen2-VL>)

Recorded evaluation configuration: {"Type":"VLMs"}. This record does not assert independent reproduction.

### PaliGemma

PR-F1: 0.2348. Result ID: 699693995698665ec4dcda0a.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65e0c.md>)

Recorded evaluation configuration: {"Type":"VLMs"}. This record does not assert independent reproduction.

### MiniCPM-V

PR-F1: 0.223. Result ID: 699693995698665ec4dcd9f7.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74681e209131cea6183d.md>)

[Implementation](<https://github.com/OpenBMB/MiniCPM-V>)

Recorded evaluation configuration: {"Model":"Individual Model","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### Router-4-L

PR-F1: 0.212. Result ID: 699693995698665ec4dcd9fb.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Implementation](<https://github.com/devichand579/MAC>)

Recorded evaluation configuration: {"Router configuration":"Router-4","Optimization goal":"L (Minimum latency)","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### Router-4-P

PR-F1: 0.212. Result ID: 699693995698665ec4dcd9fc.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Implementation](<https://github.com/devichand579/MAC>)

Recorded evaluation configuration: {"Router configuration":"Router-4","Optimization goal":"P (Maximum performance)","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### PaliGemma

PR-F1: 0.199. Result ID: 699693995698665ec4dcd9f8.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65e0c.md>)

Recorded evaluation configuration: {"Model":"Individual Model","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### Qwen2-VL

PR-F1: 0.197. Result ID: 699693995698665ec4dcd9fa.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75151e209131cea65976.md>)

[Implementation](<https://github.com/QwenLM/Qwen2-VL>)

Recorded evaluation configuration: {"Model":"Individual Model","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### Router-2-L

PR-F1: 0.196. Result ID: 699693995698665ec4dcd9fd.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Implementation](<https://github.com/devichand579/MAC>)

Recorded evaluation configuration: {"Router configuration":"Router-2","Optimization goal":"L (Minimum latency)","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### Router-2-P

PR-F1: 0.196. Result ID: 699693995698665ec4dcd9fe.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

[Implementation](<https://github.com/devichand579/MAC>)

Recorded evaluation configuration: {"Router configuration":"Router-2","Optimization goal":"P (Maximum performance)","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### QB

PR-F1: 0.1755. Result ID: 699693995698665ec4dcda08.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

Recorded evaluation configuration: {"Type":"Textual"}. This record does not assert independent reproduction.

### MPC++

PR-F1: 0.1523. Result ID: 699693995698665ec4dcda07.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

Recorded evaluation configuration: {"Type":"Textual"}. This record does not assert independent reproduction.

### QB

PR-F1: 0.135. Result ID: 699693995698665ec4dcd9f9.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

Recorded evaluation configuration: {"Model":"Individual Model","Inference engine":"vLLM"}. This record does not assert independent reproduction.

### MPC

PR-F1: 0.0845. Result ID: 699693995698665ec4dcda06.

[Reporting paper](<https://www.sota2.com/research/paper/69647fda53fdd5c3617445a9.md>)

Recorded evaluation configuration: {"Type":"Textual"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
