Multimodal Autocomplete on ImageChat (unseen prefixes)
0.281PR-F1Router-4-Max (Oracle)
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Router-4-Max (Oracle)Router configuration=Router-4, Oracle=true2026.01 | 0.281 | 0.09 | — | — | — | — | — | — | |
| MiniCPM-VType=VLMs2026.01 | 0.2586 | 0.0715 | — | 0.9892 | 0.3128 | 0.2205 | 16.1 | 0.1246 | |
| Qwen2-VLType=VLMs2026.01 | 0.2356 | 0.0577 | — | 0.9889 | 0.2931 | 0.1971 | 16.2 | 0.1422 | |
| PaliGemmaType=VLMs2026.01 | 0.2348 | 0.0616 | — | 0.9881 | 0.285 | 0.1996 | 16.7 | 0.1148 | |
| MiniCPM-VModel=Individual Model, Inference engine=vLLM2026.01 | 0.223 | 0.067 | 2.08 | — | — | — | — | — | |
| Router-4-LRouter configuration=Router-4, Optimization goal=L (Minimum latency), Inference engine=vLLM2026.01 | 0.212 | 0.056 | 0.966 | — | — | — | — | — | |
| Router-4-PRouter configuration=Router-4, Optimization goal=P (Maximum performance), Inference engine=vLLM2026.01 | 0.212 | 0.056 | 0.966 | — | — | — | — | — | |
| PaliGemmaModel=Individual Model, Inference engine=vLLM2026.01 | 0.199 | 0.057 | 1.49 | — | — | — | — | — | |
| Qwen2-VLModel=Individual Model, Inference engine=vLLM2026.01 | 0.197 | 0.053 | 0.733 | — | — | — | — | — | |
| Router-2-LRouter configuration=Router-2, Optimization goal=L (Minimum latency), Inference engine=vLLM2026.01 | 0.196 | 0.053 | 0.288 | — | — | — | — | — | |
| Router-2-PRouter configuration=Router-2, Optimization goal=P (Maximum performance), Inference engine=vLLM2026.01 | 0.196 | 0.053 | 0.288 | — | — | — | — | — | |
| QBType=Textual2026.01 | 0.1755 | 0.0373 | — | 0.9604 | 0.3065 | 0.1225 | 5.9 | 0.0955 | |
| MPC++Type=Textual2026.01 | 0.1523 | 0.0341 | — | 0.6728 | 0.208 | 0.1202 | 17.3 | 0.0371 | |
| QBModel=Individual Model, Inference engine=vLLM2026.01 | 0.135 | 0.036 | 0.001 | — | — | — | — | — | |
| MPCType=Textual2026.01 | 0.0845 | 0.0007 | — | 0.2749 | 0.112 | 0.0685 | 27.7 | 0.003 |