Multi-choice Visual Question Answering on camroll
86.4AccuracyOracle
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OracleBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=0.0h, Build Tokens=~2.0k2026.06 | 86.4 | 100 | |
| All imagesBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=0.0h, Build Tokens=~750k2026.06 | 76.5 | 100 | |
| camroll-agentBase Model=Gemini-2.5-Flash, Retrieval Embedding=all-MiniLM-L6-v2, Build Time=1.5h, Build Tokens=~3.2k2026.06 | 70.5 | 88.5 | |
| All captionsBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=1.5h, Build Tokens=~150k2026.06 | 63.4 | 100 | |
| ClaudeCodeBase Model=Sonnet-4.6, Retrieval Embedding=proprietary, unclear trace, Build Time=0.0h, Build Tokens=~59.0k2026.06 | 54 | — | |
| Mem0Base Model=Gemini-2.5-Flash, Retrieval Embedding=text-embedding-small-3, Build Time=1.5h, Build Tokens=~1.0k2026.06 | 53.2 | 75.3 | |
| LightMemBase Model=Gemini-2.5-Flash, Retrieval Embedding=all-MiniLM-L6-v2, Build Time=1.5h, Build Tokens=~1.0k2026.06 | 52.7 | 70.3 | |
| HippoRAG2Base Model=Gemini-2.5-Flash, Retrieval Embedding=NV-Embed-v2, Build Time=1.6h, Build Tokens=~1.0k2026.06 | 48.5 | 50.1 | |
| Self-RAGBase Model=LLama-2, Retrieval Embedding=contriever-msmarco, Build Time=1.5h, Build Tokens=~2.0k2026.06 | 46.2 | 25.8 | |
| SimpleMemBase Model=Gemini-2.5-Flash, Retrieval Embedding=Qwen3-Embedding-0.6B, Build Time=3.0h, Build Tokens=~0.5k2026.06 | 44.6 | 57.8 | |
| MemOSBase Model=Gemini-2.5-Flash, Retrieval Embedding=BAAI/bge-m3, Build Time=4.0h, Build Tokens=~3.1k2026.06 | 32.3 | 27.5 | |
| NothingBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=0.0h, Build Tokens=~502026.06 | 30 | 0 |