Question Answering on SimpleQA Verified
82.5AccuracyHEART
Evaluation Results
| Method | Links | |
|---|---|---|
| HEARTModel=Gemini 3 Pro2025.09 | 82.5 | |
| Self ReflectionModel=Gemini 3 Pro2025.09 | 82.4 | |
| CoTModel=Gemini 3 Pro2025.09 | 81.38 | |
| WaitModel=Gemini 3 Pro2025.09 | 80.18 | |
| Self ReflectionModel=GPT-5-nano2025.09 | 78.39 | |
| HEARTModel=GPT-5-nano2025.09 | 76.84 | |
| WaitModel=GPT-5-nano2025.09 | 76.28 | |
| CoTModel=GPT-5-nano2025.09 | 74.58 | |
| VanillaModel=Gemini 3 Pro2025.09 | 72.27 | |
| VanillaModel=GPT-5-nano2025.09 | 70.83 | |
| HEARTModel=Deepseek-Reasoner2025.09 | 68.96 | |
| WaitModel=Deepseek-Reasoner2025.09 | 64.39 | |
| HEARTModel=Gemini 3 Flash2025.09 | 64.29 | |
| CoTModel=Gemini 3 Flash2025.09 | 63.85 | |
| Self ReflectionModel=Gemini 3 Flash2025.09 | 61.39 | |
| WaitModel=Gemini 3 Flash2025.09 | 61.28 | |
| CoTModel=Deepseek-Reasoner2025.09 | 61.2 | |
| HEARTModel=Claude 4 Sonnet2025.09 | 59.39 | |
| Self ReflectionModel=Deepseek-Reasoner2025.09 | 58.84 | |
| Self ReflectionModel=Claude 4 Sonnet2025.09 | 58.29 | |
| HEARTModel=Gemini 2.5 Pro2025.09 | 57.29 | |
| CoTModel=Claude 4 Sonnet2025.09 | 56.58 | |
| WaitModel=Gemini 2.5 Pro2025.09 | 55.75 | |
| WaitModel=Claude 4 Sonnet2025.09 | 55.29 | |
| Self ReflectionModel=Gemini 2.5 Pro2025.09 | 52.38 | |
| CoTModel=Gemini 2.5 Pro2025.09 | 51.19 | |
| HEARTModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 49.88 | |
| HEARTModel=Claude-4-Sonnet (Thinking Off)2025.09 | 48.97 | |
| HEARTModel=Deepseek-Chat2025.09 | 48.14 | |
| HEARTModel=Gemini 2.5 Flash2025.09 | 47.38 | |
| Self ReflectionModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 46.3 | |
| WaitModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 46.2 | |
| WaitModel=Gemini 2.5 Flash2025.09 | 45.84 | |
| CoTModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 45.4 | |
| WaitModel=Claude-4-Sonnet (Thinking Off)2025.09 | 45.28 | |
| CoTModel=Deepseek-Chat2025.09 | 45.23 | |
| CoTModel=Claude-4-Sonnet (Thinking Off)2025.09 | 44.39 | |
| Self ReflectionModel=Deepseek-Chat2025.09 | 44.29 | |
| CoTModel=Gemini 2.5 Flash2025.09 | 44.2 | |
| WaitModel=Deepseek-Chat2025.09 | 44.19 | |
| Self ReflectionModel=Gemini 2.5 Flash2025.09 | 42.18 | |
| Self ReflectionModel=Claude-4-Sonnet (Thinking Off)2025.09 | 41.49 | |
| APEXModel=Gemini 2.5 Flash2026.06 | 35 | |
| VanillaModel=Gemini 3 Flash2025.09 | 32.77 | |
| VanillaModel=Gemini 2.5 Pro2025.09 | 32.7 | |
| VanillaModel=Deepseek-Reasoner2025.09 | 31.92 | |
| VanillaModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 29.5 | |
| VanillaModel=Deepseek-Chat2025.09 | 29.38 | |
| GEPAModel=Gemini 2.5 Flash2026.06 | 28.8 | |
| VanillaModel=Gemini 2.5 Flash2025.09 | 28.48 | |
| APO (|D|)Model=Gemini 2.5 Flash2026.06 | 27.4 | |
| APO (|D|/2)Model=Gemini 2.5 Flash2026.06 | 25 | |
| VanillaModel=Claude 4 Sonnet2025.09 | 24.98 | |
| VanillaModel=Claude-4-Sonnet (Thinking Off)2025.09 | 23.75 | |
| Initial PromptModel=Gemini 2.5 Flash2026.06 | 23.6 | |
| HEARTModel=Gemma3 12b Instruct2025.09 | 18.47 | |
| HEARTModel=Gemma3 4b Instruct2025.09 | 15.61 | |
| WaitModel=Gemma3 12b Instruct2025.09 | 14.29 | |
| CoTModel=Gemma3 12b Instruct2025.09 | 12.38 | |
| APO (|D|)Model=Gemma 3 27B2026.06 | 11.5 | |
| APEXModel=Gemma 3 27B2026.06 | 11.5 | |
| Self ReflectionModel=Gemma3 12b Instruct2025.09 | 10.29 | |
| Self ReflectionModel=Gemma3 4b Instruct2025.09 | 10 | |
| Initial PromptModel=Gemma 3 27B2026.06 | 9.4 | |
| GEPAModel=Gemma 3 27B2026.06 | 9.4 | |
| WaitModel=Gemma3 4b Instruct2025.09 | 9.04 | |
| APO (|D|/2)Model=Gemma 3 27B2026.06 | 8.4 | |
| Diff.Backbone=Qwen3-4B-Base, Synthesis Strategy=Difficulty-weighted2026.06 | 8.2 | |
| KDoSBackbone=Qwen3-4B-Base2026.06 | 8 | |
| Uni.Backbone=Qwen3-4B-Base, Synthesis Strategy=Uniform2026.06 | 7.9 | |
| CoTModel=Gemma3 4b Instruct2025.09 | 7.89 | |
| SPBackbone=Qwen3-4B-Base, Training Data=Seed Pool2026.06 | 6.8 | |
| Qual.Backbone=Qwen3-4B-Base, Synthesis Strategy=Quality-filtered2026.06 | 6.8 | |
| Rand.Backbone=Qwen3-4B-Base, Synthesis Strategy=Random2026.06 | 6.7 | |
| VanillaModel=Gemma3 12b Instruct2025.09 | 5.83 | |
| VanillaModel=Gemma3 4b Instruct2025.09 | 4.7 | |
| BaseBackbone=Qwen3-4B-Base2026.06 | 3.6 |