Free-form Visual Question Answering on camroll
6.33Judge ScoreOracle
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OracleBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=0.0h, Build Tokens=~2.0k2026.06 | 6.33 | 100 | |
| All imagesBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=0.0h, Build Tokens=~750k2026.06 | 5.01 | 100 | |
| camroll-agentBase Model=Gemini-2.5-Flash, Retrieval Embedding=all-MiniLM-L6-v2, Build Time=1.5h, Build Tokens=~3.2k2026.06 | 4.11 | 83.1 | |
| All captionsBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=1.5h, Build Tokens=~150k2026.06 | 3.82 | 100 | |
| ClaudeCodeBase Model=Sonnet-4.6, Retrieval Embedding=proprietary, unclear trace, Build Time=0.0h, Build Tokens=~59.0k2026.06 | 3.77 | — | |
| Mem0Base Model=Gemini-2.5-Flash, Retrieval Embedding=text-embedding-small-3, Build Time=1.5h, Build Tokens=~1.0k2026.06 | 2.68 | 75.3 | |
| HippoRAG2Base Model=Gemini-2.5-Flash, Retrieval Embedding=NV-Embed-v2, Build Time=1.6h, Build Tokens=~1.0k2026.06 | 2.58 | 50.1 | |
| LightMemBase Model=Gemini-2.5-Flash, Retrieval Embedding=all-MiniLM-L6-v2, Build Time=1.5h, Build Tokens=~1.0k2026.06 | 2.44 | 70.2 | |
| Self-RAGBase Model=LLama-2, Retrieval Embedding=contriever-msmarco, Build Time=1.5h, Build Tokens=~2.0k2026.06 | 2.41 | 19.8 | |
| SimpleMemBase Model=Gemini-2.5-Flash, Retrieval Embedding=Qwen3-Embedding-0.6B, Build Time=3.0h, Build Tokens=~0.5k2026.06 | 1.7 | 58.6 | |
| MemOSBase Model=Gemini-2.5-Flash, Retrieval Embedding=BAAI/bge-m3, Build Time=4.0h, Build Tokens=~3.1k2026.06 | 1.09 | 27.3 | |
| NothingBase Model=Gemini-2.5-Flash, Retrieval Embedding=no retrieval step, Build Time=0.0h, Build Tokens=~502026.06 | 0 | 0 |