Logical Reasoning on SLR-BENCH Extended Leaderboard
15.5LRL Scoreo3
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| o3Reasoning enabled=true, Total Compute Tok. (M)=4.30, Total Compute Costs ($)=207.242025.06 | 15.5 | 80 | 99 | 93 | 74 | 45 | |
| gpt-5Reasoning enabled=true, Total Compute Tok. (M)=16.40, Total Compute Costs ($)=103.132025.06 | 15.4 | 98 | 100 | 90 | 72 | 46 | |
| gpt-5-mini-highReasoning enabled=true, Total Compute Tok. (M)=13.12, Total Compute Costs ($)=27.982025.06 | 14.2 | 94 | 99 | 83 | 63 | 38 | |
| o4-mini-highReasoning enabled=true, Total Compute Tok. (M)=4.62, Total Compute Costs ($)=24.242025.06 | 12.8 | 88 | 98 | 96 | 40 | 21 | |
| o4-miniReasoning enabled=true, Total Compute Tok. (M)=3.98, Total Compute Costs ($)=21.432025.06 | 12.3 | 86 | 93 | 88 | 52 | 13 | |
| gpt-5-miniReasoning enabled=true, Total Compute Tok. (M)=4.90, Total Compute Costs ($)=11.542025.06 | 12 | 95 | 99 | 82 | 41 | 20 | |
| o1Reasoning enabled=true, Total Compute Tok. (M)=5.19, Total Compute Costs ($)=364.722025.06 | 11.9 | 68 | 92 | 89 | 41 | 15 | |
| o3-miniReasoning enabled=true, Total Compute Tok. (M)=4.73, Total Compute Costs ($)=24.712025.06 | 11.6 | 75 | 97 | 90 | 37 | 7 | |
| Qwen3-235B-A22B-Thinking-2507Reasoning enabled=true, Total Compute Tok. (M)=26.32, Total Compute Costs ($)=15.532025.06 | 10.4 | 89 | 82 | 76 | 38 | 12 | |
| o4-mini-lowReasoning enabled=true, Total Compute Tok. (M)=0.77, Total Compute Costs ($)=7.262025.06 | 10.3 | 91 | 91 | 81 | 25 | 9 | |
| Qwen3-235B-A22B-Instruct-2507Reasoning enabled=false, Total Compute Tok. (M)=5.81, Total Compute Costs ($)=4.452025.06 | 10.3 | 100 | 99 | 76 | 20 | 10 | |
| o1-miniReasoning enabled=true, Total Compute Tok. (M)=3.65, Total Compute Costs ($)=19.982025.06 | 10.1 | 95 | 97 | 82 | 20 | 3 | |
| gpt-5-mini-lowReasoning enabled=true, Total Compute Tok. (M)=1.17, Total Compute Costs ($)=4.072025.06 | 9.7 | 88 | 97 | 71 | 20 | 7 | |
| Qwen3-Next-80B-A3B-ThinkingReasoning enabled=true, Total Compute Tok. (M)=16.03, Total Compute Costs ($)=20.332025.06 | 9.6 | 96 | 93 | 73 | 18 | 8 | |
| Qwen3-32BReasoning enabled=true, Total Compute Tok. (M)=7.73, Total Compute Costs ($)=1.912025.06 | 9 | 100 | 97 | 64 | 14 | 6 | |
| Qwen3-30B-A3B-Thinking-2507Reasoning enabled=true, Total Compute Tok. (M)=17.52, Total Compute Costs ($)=4.292025.06 | 9 | 97 | 97 | 67 | 14 | 3 | |
| Qwen3-VL-30B-A3B-ThinkingReasoning enabled=true, Total Compute Tok. (M)=18.07, Total Compute Costs ($)=11.942025.06 | 8.9 | 83 | 94 | 70 | 12 | 3 | |
| Qwen3-Next-80B-A3B-InstructReasoning enabled=false, Total Compute Tok. (M)=8.68, Total Compute Costs ($)=7.682025.06 | 8.8 | 100 | 93 | 64 | 15 | 5 | |
| R1-Llama-70BReasoning enabled=true, Total Compute Tok. (M)=11.61, Total Compute Costs ($)=5.332025.06 | 8.8 | 75 | 98 | 67 | 8 | 4 | |
| Gemini-thinkingReasoning enabled=true2025.06 | 8.6 | 83 | 93 | 65 | 13 | 1 | |
| gpt-5-nanoReasoning enabled=true, Total Compute Tok. (M)=6.17, Total Compute Costs ($)=2.812025.06 | 8.5 | 99 | 97 | 61 | 10 | 2 | |
| Qwen3-VL-32B-ThinkingReasoning enabled=true, Total Compute Tok. (M)=19.33, Total Compute Costs ($)=23.822025.06 | 8.4 | 99 | 86 | 66 | 14 | 3 | |
| Qwen3-4B-Thinking-2507Reasoning enabled=true, Total Compute Tok. (M)=12.71, Total Compute Costs ($)=0.432025.06 | 8.1 | 99 | 98 | 56 | 8 | 2 | |
| Qwen3-30B-A3B-Instruct-2507Reasoning enabled=false, Total Compute Tok. (M)=7.46, Total Compute Costs ($)=2.082025.06 | 8 | 100 | 95 | 57 | 5 | 3 | |
| Qwen3-30B-A3BReasoning enabled=true, Total Compute Tok. (M)=8.62, Total Compute Costs ($)=2.332025.06 | 7.9 | 100 | 93 | 57 | 7 | 1 | |
| Qwen3-VL-8B-ThinkingReasoning enabled=true, Total Compute Tok. (M)=20.77, Total Compute Costs ($)=10.442025.06 | 7.7 | 97 | 87 | 55 | 9 | 2 | |
| Qwen3-8BReasoning enabled=true, Total Compute Tok. (M)=10.17, Total Compute Costs ($)=1.662025.06 | 7.6 | 100 | 93 | 50 | 7 | 3 | |
| Qwen3-VL-4B-ThinkingReasoning enabled=true, Total Compute Tok. (M)=0.00, Total Compute Costs ($)=0.002025.06 | 7.6 | 86 | 91 | 52 | 6 | 2 | |
| Qwen3-14BReasoning enabled=true, Total Compute Tok. (M)=8.99, Total Compute Costs ($)=2.342025.06 | 7.5 | 100 | 88 | 53 | 8 | 2 | |
| gpt-4.5-prevReasoning enabled=false, Total Compute Tok. (M)=0.37, Total Compute Costs ($)=576.402025.06 | 7.3 | 100 | 94 | 47 | 5 | 1 | |
| Qwen3-4BReasoning enabled=true, Total Compute Tok. (M)=9.77, Total Compute Costs ($)=0.352025.06 | 6.9 | 99 | 91 | 41 | 5 | 1 | |
| Qwen3-4B-Instruct-2507Reasoning enabled=false, Total Compute Tok. (M)=9.40, Total Compute Costs ($)=0.342025.06 | 6.6 | 100 | 73 | 50 | 6 | 2 | |
| Qwen3-235B-A22BReasoning enabled=false, Total Compute Tok. (M)=1.05, Total Compute Costs ($)=1.882025.06 | 6.5 | 100 | 87 | 41 | 3 | 0 | |
| gpt-4oReasoning enabled=false, Total Compute Tok. (M)=0.26, Total Compute Costs ($)=20.032025.06 | 6.2 | 100 | 93 | 29 | 2 | 0 | |
| Qwen3-32BReasoning enabled=false, Total Compute Tok. (M)=0.63, Total Compute Costs ($)=0.492025.06 | 5.9 | 100 | 89 | 28 | 2 | 0 | |
| Qwen3-14BReasoning enabled=false, Total Compute Tok. (M)=0.90, Total Compute Costs ($)=0.562025.06 | 5.6 | 100 | 86 | 25 | 2 | 0 | |
| Llama 3.3-70BReasoning enabled=false, Total Compute Tok. (M)=0.49, Total Compute Costs ($)=0.822025.06 | 5.6 | 100 | 90 | 22 | 1 | 0 | |
| gpt-4-turboReasoning enabled=false, Total Compute Tok. (M)=0.41, Total Compute Costs ($)=81.302025.06 | 5.4 | 100 | 89 | 18 | 2 | 0 | |
| Qwen3-Coder-30B-A3B-InstructReasoning enabled=false, Total Compute Tok. (M)=3.32, Total Compute Costs ($)=1.272025.06 | 5.3 | 100 | 88 | 17 | 2 | 0 | |
| Qwen3-30B-A3BReasoning enabled=false, Total Compute Tok. (M)=1.31, Total Compute Costs ($)=0.732025.06 | 5.2 | 100 | 85 | 19 | 1 | 0 | |
| Qwen3-8BReasoning enabled=false, Total Compute Tok. (M)=1.57, Total Compute Costs ($)=0.472025.06 | 4.6 | 100 | 78 | 15 | 0 | 0 | |
| Qwen3-4BReasoning enabled=false, Total Compute Tok. (M)=1.47, Total Compute Costs ($)=0.152025.06 | 4.1 | 100 | 70 | 12 | 1 | 0 | |
| Llama 3.1-8BReasoning enabled=false, Total Compute Tok. (M)=0.20, Total Compute Costs ($)=0.142025.06 | 3.8 | 80 | 70 | 7 | 0 | 0 | |
| Qwen3-VL-30B-A3B-InstructReasoning enabled=false, Total Compute Tok. (M)=8.21, Total Compute Costs ($)=1.122025.06 | 3.7 | 100 | 71 | 4 | 0 | 0 | |
| Qwen3-VL-32B-InstructReasoning enabled=false, Total Compute Tok. (M)=8.21, Total Compute Costs ($)=2.622025.06 | 3.5 | 100 | 68 | 2 | 0 | 0 | |
| Qwen3-VL-8B-InstructReasoning enabled=false, Total Compute Tok. (M)=8.21, Total Compute Costs ($)=0.332025.06 | 3.3 | 99 | 65 | 1 | 0 | 0 | |
| Qwen3-1.7BReasoning enabled=true, Total Compute Tok. (M)=8.21, Total Compute Costs ($)=0.312025.06 | 1.9 | 100 | 35 | 4 | 0 | 0 | |
| Qwen3-VL-2B-InstructReasoning enabled=false, Total Compute Tok. (M)=0.00, Total Compute Costs ($)=0.002025.06 | 1.8 | 58 | 36 | 0 | 0 | 0 | |
| Qwen3-0.6BReasoning enabled=false, Total Compute Tok. (M)=4.45, Total Compute Costs ($)=0.222025.06 | 1.5 | 80 | 30 | 0 | 0 | 0 | |
| Llama 3.2-3BReasoning enabled=false, Total Compute Tok. (M)=0.10, Total Compute Costs ($)=0.112025.06 | 1 | 25 | 19 | 1 | 0 | 0 | |
| Qwen3-VL-2B-ThinkingReasoning enabled=true, Total Compute Tok. (M)=0.00, Total Compute Costs ($)=0.002025.06 | 0.4 | 87 | 8 | 0 | 0 | 0 | |
| Qwen3-1.7BReasoning enabled=false, Total Compute Tok. (M)=1.59, Total Compute Costs ($)=0.152025.06 | 0.3 | 95 | 6 | 1 | 0 | 0 | |
| Qwen3-0.6BReasoning enabled=true, Total Compute Tok. (M)=5.57, Total Compute Costs ($)=0.252025.06 | 0.2 | 98 | 4 | 0 | 0 | 0 | |
| Llama 3.2-1BReasoning enabled=false, Total Compute Tok. (M)=0.47, Total Compute Costs ($)=0.122025.06 | 0 | 93 | 0 | 0 | 0 | 0 |