Logical Reasoning on Formal Logic
87.8AccuracyUAB
Evaluation Results
| Method | Links | |
|---|---|---|
| UABBackbone Model=GPT-OSS-20B, Budget=N=42026.05 | 87.8 | |
| UABBackbone Model=Cohere, Budget=N=42026.05 | 87.3 | |
| UniformBackbone Model=GPT-OSS-20B, Budget=N=42026.05 | 86.5 | |
| LLM-JudgeBackbone Model=GPT-OSS-20B, Budget=N=42026.05 | 86.5 | |
| LLM-JudgeBackbone Model=Cohere, Budget=N=42026.05 | 86.1 | |
| UniformBackbone Model=Cohere, Budget=N=42026.05 | 84.3 | |
| UABBackbone Model=Gemma3-27B, Budget=N=42026.05 | 84.1 | |
| LLM-JudgeBackbone Model=Gemma3-27B, Budget=N=42026.05 | 83.9 | |
| UniformBackbone Model=Gemma3-27B, Budget=N=42026.05 | 82.5 | |
| N=1Backbone Model=Cohere, Budget=N=12026.05 | 81.4 | |
| N=1Backbone Model=Gemma3-27B, Budget=N=12026.05 | 78 | |
| OracleModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 77.8 | |
| N=1Backbone Model=GPT-OSS-20B, Budget=N=12026.05 | 76.1 | |
| OracleModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 74.1 | |
| OracleModel=Qwen2.5-3B, Black-Box Friendly=true2026.04 | 73.5 | |
| OracleModel=Llama3.2-3B, Black-Box Friendly=true2026.04 | 71.4 | |
| OracleModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 67.7 | |
| DARModel=Falcon3-7B2026.03 | 60.3 | |
| Uncertain PromptBackbone Model=Falcon3-7B, Number of Agents=4, Number of Rounds=22026.03 | 60.3 | |
| UABModel=Qwen2.5-7B, N=42026.05 | 59.2 | |
| DARBackbone Model=Falcon3-7B, Number of Agents=4, Number of Rounds=22026.03 | 59 | |
| MAD-M2Model=Falcon3-7B2026.03 | 58.7 | |
| Society Of MindBackbone Model=Falcon3-7B, Number of Agents=4, Number of Rounds=22026.03 | 57.7 | |
| Society Of MindModel=Falcon3-7B2026.03 | 57.5 | |
| Uncertain PromptModel=Falcon3-7B2026.03 | 57.5 | |
| Vote PromptModel=Falcon3-7B2026.03 | 57.5 | |
| Vote PromptBackbone Model=Falcon3-7B, Number of Agents=4, Number of Rounds=22026.03 | 57.4 | |
| LLM-JudgeModel=Qwen2.5-7B, N=42026.05 | 56.9 | |
| UniformModel=Qwen2.5-7B, N=42026.05 | 56.5 | |
| RandomModel=Qwen2.5-7B, N=42026.05 | 56.2 | |
| N=1Model=Qwen2.5-7B, N=12026.05 | 55.7 | |
| MAD-M^2Backbone Model=Falcon3-7B, Number of Agents=4, Number of Rounds=22026.03 | 55.1 | |
| LengthModel=Qwen2.5-7B, N=42026.05 | 54 | |
| RCSuniModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 52.9 | |
| RCSmedoidModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 52.6 | |
| Uncertain PromptModel=Llama3.1-8B2026.03 | 52.4 | |
| Vote PromptModel=Llama3.1-8B2026.03 | 52.4 | |
| GreedyModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 52.4 | |
| RCSprobModel=Gemma2-9B, Black-Box Friendly=false2026.04 | 52.4 | |
| CEModel=Gemma2-9B, Black-Box Friendly=false2026.04 | 52.1 | |
| DARModel=Llama3.1-8B2026.03 | 52 | |
| RCSfreqModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 51.8 | |
| Vote PromptBackbone Model=Llama3.1-8B, Number of Agents=4, Number of Rounds=22026.03 | 51.6 | |
| SCModel=Gemma2-9B, Black-Box Friendly=true2026.04 | 51.6 | |
| Society Of MindModel=Llama3.1-8B2026.03 | 50 | |
| ANLLModel=Gemma2-9B, Black-Box Friendly=false2026.04 | 50 | |
| DARBackbone Model=Llama3.1-8B, Number of Agents=4, Number of Rounds=22026.03 | 49.7 | |
| Majority VoteBackbone Model=Falcon3-7B, Number of Agents=4, Number of Rounds=22026.03 | 49.4 | |
| RCSmedoidModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 49.2 | |
| RCSuniModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 49.2 | |
| RCSprobModel=Qwen2.5-7B, Black-Box Friendly=false2026.04 | 49.2 | |
| Uncertain PromptBackbone Model=Llama3.1-8B, Number of Agents=4, Number of Rounds=22026.03 | 48.8 | |
| Majority VoteModel=Falcon3-7B2026.03 | 48.7 | |
| Society Of MindBackbone Model=Llama3.1-8B, Number of Agents=4, Number of Rounds=22026.03 | 48.7 | |
| CEModel=Qwen2.5-7B, Black-Box Friendly=false2026.04 | 48.7 | |
| SCModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 48.4 | |
| NLLModel=Gemma2-9B, Black-Box Friendly=false2026.04 | 48.1 | |
| RCSfreqModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 47.9 | |
| Single-agentModel=Llama3.1-8B2026.03 | 47.2 | |
| UABModel=Qwen2.5-1.5B, N=42026.05 | 47.1 | |
| LLM-JudgeModel=Qwen2.5-1.5B, N=42026.05 | 45.8 | |
| Single-agentBackbone Model=Llama3.1-8B2026.03 | 45.6 | |
| Majority VoteBackbone Model=Llama3.1-8B, Number of Agents=4, Number of Rounds=22026.03 | 45.6 | |
| Majority VoteModel=Llama3.1-8B2026.03 | 45.5 | |
| RCSuniModel=Qwen2.5-3B, Black-Box Friendly=true2026.04 | 44.7 | |
| RCSprobModel=Qwen2.5-3B, Black-Box Friendly=false2026.04 | 43.9 | |
| RCSmedoidModel=Qwen2.5-3B, Black-Box Friendly=true2026.04 | 43.6 | |
| GreedyModel=Qwen2.5-7B, Black-Box Friendly=true2026.04 | 43.6 | |
| DARBackbone Model=Qwen2.5-3B, Number of Agents=4, Number of Rounds=22026.03 | 43.4 | |
| RCSfreqModel=Qwen2.5-3B, Black-Box Friendly=true2026.04 | 43.4 | |
| LengthModel=Qwen2.5-1.5B, N=42026.05 | 43.4 | |
| UABModel=Llama3.2-3B, N=42026.05 | 43.4 | |
| Uncertain PromptModel=Qwen2.5-3B2026.03 | 42.5 | |
| Single-agentBackbone Model=Falcon3-7B2026.03 | 42.5 | |
| MAD-M2Model=Qwen2.5-3B2026.03 | 42.3 | |
| Majority VoteModel=Qwen2.5-3B2026.03 | 42.3 | |
| UniformModel=Qwen2.5-1.5B, N=42026.05 | 42.1 | |
| N=1Model=Qwen2.5-1.5B, N=12026.05 | 41.7 | |
| CEModel=Qwen2.5-3B, Black-Box Friendly=false2026.04 | 41.3 | |
| UniformModel=Llama3.2-3B, N=42026.05 | 41.3 | |
| Vote PromptModel=Qwen2.5-3B2026.03 | 40.9 | |
| DARModel=Qwen2.5-3B2026.03 | 40.9 | |
| Single-agentModel=Falcon3-7B2026.03 | 40.9 | |
| RandomModel=Qwen2.5-1.5B, N=42026.05 | 40.9 | |
| SCModel=Qwen2.5-3B, Black-Box Friendly=true2026.04 | 40.5 | |
| Majority VoteBackbone Model=Qwen2.5-3B, Number of Agents=4, Number of Rounds=22026.03 | 40.1 | |
| LengthModel=Llama3.2-3B, N=42026.05 | 39.7 | |
| Vote PromptBackbone Model=Qwen2.5-3B, Number of Agents=4, Number of Rounds=22026.03 | 39.4 | |
| RCSuniModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 39.4 | |
| Society Of MindModel=Qwen2.5-3B2026.03 | 39.3 | |
| MAD-M^2Backbone Model=Llama3.1-8B, Number of Agents=4, Number of Rounds=22026.03 | 39.2 | |
| LLM-JudgeModel=Llama3.2-3B, N=42026.05 | 39.2 | |
| RandomModel=Llama3.2-3B, N=42026.05 | 38.8 | |
| Society Of MindBackbone Model=Qwen2.5-3B, Number of Agents=4, Number of Rounds=22026.03 | 38.6 | |
| RCSmedoidModel=Llama3.1-8B, Black-Box Friendly=true2026.04 | 37.8 | |
| Vote PromptModel=Qwen2.5-1.5B2026.03 | 37.7 | |
| DARModel=Qwen2.5-1.5B2026.03 | 37.3 | |
| N=1Model=Llama3.2-3B, N=12026.05 | 37.2 | |
| RCSprobModel=Llama3.1-8B, Black-Box Friendly=false2026.04 | 36.8 | |
| DARBackbone Model=Qwen2.5-1.5B, Number of Agents=4, Number of Rounds=22026.03 | 36.2 |