Reasoning on MMLU-Pro (Subject Averages)
55Average ScoreRAT
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| RATBackbone=GPT-4o-mini, Category=Advanced RAG Methods2025.08 | 55 | 57.5 | 55.3 | 73 | 34.2 | |
| CoTBackbone=GPT-4o-mini2025.08 | 54.5 | 56.7 | 53.1 | 70.4 | 38.2 | |
| Standard RAGBackbone=GPT-4o-mini2025.08 | 53.5 | 57 | 52.3 | 68.6 | 36.2 | |
| Search-o1Backbone=GPT-4o-mini, Category=Advanced RAG Methods2025.08 | 53.5 | 53.5 | 55.3 | 69.8 | 35.4 | |
| UR2 (GPT Summary)Backbone=Qwen-2.5-7B2025.08 | 53.3 | 53.2 | 53 | 72.2 | 35 | |
| Self-AskBackbone=GPT-4o-mini, Category=Advanced RAG Methods2025.08 | 51 | 56.3 | 48.5 | 67.8 | 31.2 | |
| UR2 (qw Summary)Backbone=Qwen-2.5-7B2025.08 | 50.3 | 51.2 | 49.7 | 67.8 | 32.6 | |
| Open-Reasoner-ZeroBackbone=Qwen-2.5-7B, Category=CoT-RL Methods2025.08 | 49.6 | 50 | 46.6 | 67.5 | 34.2 | |
| Vanilla RLBackbone=Qwen-2.5-7B2025.08 | 48.4 | 52.2 | 43.5 | 64 | 33.8 | |
| UR2 (GPT summary)Backbone=Qwen-2.5-3B2025.08 | 47.8 | 47.8 | 49.3 | 63.9 | 30 | |
| UR2Model=Qwen-2.5-3B2025.08 | 47.8 | 47.8 | 49.3 | 63.9 | 30 | |
| General ReasonerBackbone=Qwen-2.5-7B, Category=CoT-RL Methods2025.08 | 47.1 | 47.9 | 44.2 | 65.9 | 30.4 | |
| RATModel=Qwen-2.5-7B2025.08 | 46.6 | 47.2 | 44.7 | 64.4 | 30 | |
| Search-o1Model=Qwen-2.5-7B2025.08 | 45.4 | 42.8 | 45.9 | 63.2 | 29.6 | |
| NeuFSModel=Qwen3-4B, InfoType=Neuron, Aggregated Shots=5, 10, 20, 302026.07 | 45.2 | — | — | — | — | |
| CoTBackbone=Qwen-2.5-7B2025.08 | 44.5 | 42.3 | 45.7 | 63.4 | 26.6 | |
| TypiClust (S.)Model=Qwen3-4B, InfoType=S., Aggregated Shots=5, 10, 20, 302026.07 | 43.7 | — | — | — | — | |
| UR2 (GPT summary)Backbone=LLaMA-3.1-8B2025.08 | 43.4 | 48.3 | 38.6 | 58 | 28.8 | |
| UR2Model=LLaMA-3.1-8B2025.08 | 43.4 | 48.3 | 38.6 | 58 | 28.8 | |
| UR2 (qw summary)Backbone=Qwen-2.5-3B2025.08 | 43.2 | 43 | 42.6 | 60.6 | 26.6 | |
| Entropy (Highest)Model=Qwen3-4B, InfoType=Highest, Aggregated Shots=5, 10, 20, 302026.07 | 43 | — | — | — | — | |
| Standard RAGBackbone=Qwen-2.5-7B2025.08 | 42.4 | 44.6 | 41.1 | 57.8 | 26 | |
| UR2 (qw summary)Backbone=LLaMA-3.1-8B2025.08 | 42.4 | 43.6 | 41.2 | 56.8 | 28 | |
| Self-AskModel=Qwen-2.5-7B2025.08 | 42.3 | 40.7 | 42.1 | 60 | 26.2 | |
| FastVoteK (S.+L.)Model=Qwen3-4B, InfoType=S.+L., Aggregated Shots=5, 10, 20, 302026.07 | 42 | — | — | — | — | |
| Entropy (Diverse)Model=Qwen3-4B, InfoType=Diverse, Aggregated Shots=5, 10, 20, 302026.07 | 41.4 | — | — | — | — | |
| RandomModel=Qwen3-4B, InfoType=N/A, Aggregated Shots=5, 10, 20, 302026.07 | 41.2 | — | — | — | — | |
| CoTBackbone=LLaMA-3.1-8B2025.08 | 40.3 | 37.8 | 40.9 | 53.4 | 29 | |
| CoTModel=LLaMA-3.1-8B2025.08 | 40.3 | 37.8 | 40.9 | 53.4 | 29 | |
| RATModel=LLaMA-3.1-8B2025.08 | 40.3 | 42.3 | 37.7 | 52.6 | 28.6 | |
| Vanilla RLBackbone=LLaMA-3.1-8B2025.08 | 40.2 | 44.6 | 36.9 | 53 | 26.4 | |
| Vanilla RLModel=LLaMA-3.1-8B2025.08 | 40.2 | 44.6 | 36.9 | 53 | 26.4 | |
| FastVoteK (L.)Model=Qwen3-4B, InfoType=L., Aggregated Shots=5, 10, 20, 302026.07 | 40.1 | — | — | — | — | |
| VoteK (S.+E.)Model=Qwen3-4B, InfoType=S.+E., Aggregated Shots=5, 10, 20, 302026.07 | 40.1 | — | — | — | — | |
| TypiClust (S.+L.)Model=Qwen3-4B, InfoType=S.+L., Aggregated Shots=5, 10, 20, 302026.07 | 39.8 | — | — | — | — | |
| PatronModel=Qwen3-4B, InfoType=S.+E., Aggregated Shots=5, 10, 20, 302026.07 | 39.1 | — | — | — | — | |
| Standard RAGBackbone=LLaMA-3.1-8B2025.08 | 38.8 | 43.6 | 33.9 | 51 | 26.6 | |
| Standard RAGModel=LLaMA-3.1-8B2025.08 | 38.8 | 43.6 | 33.9 | 51 | 26.6 | |
| Vanilla RLBackbone=Qwen-2.5-3B2025.08 | 38.7 | 40.7 | 34.7 | 55 | 24.6 | |
| Vanilla RLModel=Qwen-2.5-3B2025.08 | 38.7 | 40.7 | 34.7 | 55 | 24.6 | |
| FastVoteK (S.)Model=Qwen3-4B, InfoType=S., Aggregated Shots=5, 10, 20, 302026.07 | 38.6 | — | — | — | — | |
| SimpleRL-ZooBackbone=Qwen-2.5-7B, Category=CoT-RL Methods2025.08 | 38.3 | 35.7 | 36.9 | 55.2 | 25.4 | |
| TypiClust (L.)Model=Qwen3-4B, InfoType=L., Aggregated Shots=5, 10, 20, 302026.07 | 37.6 | — | — | — | — | |
| VoteK (S.+E.+L.)Model=Qwen3-4B, InfoType=S.+E.+L., Aggregated Shots=5, 10, 20, 302026.07 | 37.4 | — | — | — | — | |
| Standard RAGBackbone=Qwen-2.5-3B2025.08 | 37.1 | 37.8 | 36.5 | 51.4 | 23.2 | |
| Standard RAGModel=Qwen-2.5-3B2025.08 | 37.1 | 37.8 | 36.5 | 51.4 | 23.2 | |
| Self-AskModel=LLaMA-3.1-8B2025.08 | 35.6 | 39.8 | 32.1 | 47 | 23.4 | |
| RATModel=Qwen-2.5-3B2025.08 | 35.1 | 37.8 | 33.5 | 50.6 | 18.4 | |
| Search-o1Model=Qwen-2.5-3B2025.08 | 34.9 | 33.9 | 34.5 | 50.6 | 20.6 | |
| Search-o1Model=LLaMA-3.1-8B2025.08 | 34.8 | 32.6 | 32.5 | 46 | 28 | |
| CoTBackbone=Qwen-2.5-3B2025.08 | 33.8 | 33.6 | 32.3 | 48.8 | 20.6 | |
| CoTModel=Qwen-2.5-3B2025.08 | 33.8 | 33.6 | 32.3 | 48.8 | 20.6 | |
| Self-AskModel=Qwen-2.5-3B2025.08 | 32 | 33.1 | 30.5 | 44.2 | 20.2 | |
| NeuFSModel=Llama-3.2-3B, InfoType=Neuron, Aggregated Shots=5, 10, 20, 302026.07 | 25.1 | — | — | — | — | |
| RandomModel=Llama-3.2-3B, InfoType=N/A, Aggregated Shots=5, 10, 20, 302026.07 | 24.9 | — | — | — | — | |
| TypiClust (L.)Model=Llama-3.2-3B, InfoType=L., Aggregated Shots=5, 10, 20, 302026.07 | 24.8 | — | — | — | — | |
| VoteK (S.+E.)Model=Llama-3.2-3B, InfoType=S.+E., Aggregated Shots=5, 10, 20, 302026.07 | 24.6 | — | — | — | — | |
| Entropy (Highest)Model=Llama-3.2-3B, InfoType=Highest, Aggregated Shots=5, 10, 20, 302026.07 | 24.5 | — | — | — | — | |
| FastVoteK (S.)Model=Llama-3.2-3B, InfoType=S., Aggregated Shots=5, 10, 20, 302026.07 | 24.5 | — | — | — | — | |
| PatronModel=Llama-3.2-3B, InfoType=S.+E., Aggregated Shots=5, 10, 20, 302026.07 | 24.4 | — | — | — | — | |
| Entropy (Diverse)Model=Llama-3.2-3B, InfoType=Diverse, Aggregated Shots=5, 10, 20, 302026.07 | 24.4 | — | — | — | — | |
| TypiClust (S.+L.)Model=Llama-3.2-3B, InfoType=S.+L., Aggregated Shots=5, 10, 20, 302026.07 | 24.3 | — | — | — | — | |
| TypiClust (S.)Model=Llama-3.2-3B, InfoType=S., Aggregated Shots=5, 10, 20, 302026.07 | 24.2 | — | — | — | — | |
| FastVoteK (L.)Model=Llama-3.2-3B, InfoType=L., Aggregated Shots=5, 10, 20, 302026.07 | 24.2 | — | — | — | — | |
| FastVoteK (S.+L.)Model=Llama-3.2-3B, InfoType=S.+L., Aggregated Shots=5, 10, 20, 302026.07 | 24.2 | — | — | — | — | |
| VoteK (S.+E.+L.)Model=Llama-3.2-3B, InfoType=S.+E.+L., Aggregated Shots=5, 10, 20, 302026.07 | 23.3 | — | — | — | — |