Question Answering on SimpleQA (Accuracy, Conditional Accuracy)
73.1AccuracyFrontier
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FrontierModel=Gemini 3.1 Pro2025.11 | 73.1 | — | |
| ReasonerModel=Gemini 3.1 Pro2025.11 | 65.7 | — | |
| ConvFillBase Model=SmolLM2 1.7B, Talker=Gemini 3.1 Pro2025.11 | 65.1 | 98.6 | |
| ConvFillBase Model=Qwen3 0.6B, Talker=Gemini 3.1 Pro2025.11 | 64.8 | 98.5 | |
| ConvFillBase Model=Gemma 3 1B, Talker=Gemini 3.1 Pro2025.11 | 64.3 | 97.6 | |
| ConvFillBase Model=Llama 3.2 1B, Talker=Gemini 3.1 Pro2025.11 | 63.7 | 96.5 | |
| ConvFillBase Model=SmolLM2 360M, Talker=Gemini 3.1 Pro2025.11 | 62.8 | 94.8 | |
| FrontierModel=GPT-5.52025.11 | 62.7 | — | |
| ConvFillBase Model=Gemma 3 270M, Talker=Gemini 3.1 Pro2025.11 | 61.9 | 93.6 | |
| ReasonerModel=GPT-5.52025.11 | 61.3 | — | |
| ConvFillBase Model=SmolLM2 1.7B, Talker=GPT-5.52025.11 | 61 | 98.7 | |
| ConvFillBase Model=SmolLM2 135M, Talker=Gemini 3.1 Pro2025.11 | 60.7 | 91.9 | |
| ConvFillBase Model=Qwen3 0.6B, Talker=GPT-5.52025.11 | 60.3 | 97.6 | |
| ConvFillBase Model=Gemma 3 1B, Talker=GPT-5.52025.11 | 59.5 | 96.2 | |
| ConvFillBase Model=SmolLM2 360M, Talker=GPT-5.52025.11 | 57.7 | 93.3 | |
| ConvFillBase Model=Llama 3.2 1B, Talker=GPT-5.52025.11 | 57.7 | 93.5 | |
| ConvFillBase Model=Gemma 3 270M, Talker=GPT-5.52025.11 | 56.6 | 91.2 | |
| ConvFillBase Model=SmolLM2 135M, Talker=GPT-5.52025.11 | 55.2 | 89.1 | |
| FrontierModel=Claude Opus 4.72025.11 | 44.9 | — | |
| ReasonerModel=Claude Opus 4.72025.11 | 42.9 | — | |
| ConvFillBase Model=SmolLM2 1.7B, Talker=Claude Opus 4.72025.11 | 42.5 | 97.2 | |
| ConvFillBase Model=Qwen3 0.6B, Talker=Claude Opus 4.72025.11 | 42.4 | 97.2 | |
| ConvFillBase Model=Gemma 3 1B, Talker=Claude Opus 4.72025.11 | 41.5 | 96 | |
| ConvFillBase Model=Llama 3.2 1B, Talker=Claude Opus 4.72025.11 | 41.3 | 94.9 | |
| ConvFillBase Model=SmolLM2 360M, Talker=Claude Opus 4.72025.11 | 41.2 | 94.2 | |
| ConvFillBase Model=Gemma 3 270M, Talker=Claude Opus 4.72025.11 | 40.6 | 93 | |
| ConvFillBase Model=SmolLM2 135M, Talker=Claude Opus 4.72025.11 | 38 | 87.4 | |
| Base SLMModel=SmolLM2 1.7B2025.11 | 2.5 | — | |
| Base SLMModel=Gemma 3 1B2025.11 | 2.2 | — | |
| Base SLMModel=Qwen3 0.6B2025.11 | 1.4 | — | |
| Base SLMModel=SmolLM2 360M2025.11 | 1.3 | — | |
| Base SLMModel=Gemma 3 270M2025.11 | 0.9 | — | |
| Base SLMModel=Llama 3.2 1B2025.11 | 0.5 | — | |
| Base SLMModel=SmolLM2 135M2025.11 | 0.4 | — |