List your company
ResearchFormerly WizWand

Logical Reasoning on SLR-BENCH (test)

11.3LRLllama-3.1-8b-SLR
-0.4522.5995.658.701Jun 18, 2025
Updated 5mo ago

Evaluation results

MethodLinks
llama-3.1-8b-SLRPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.05, Cost ($)=0.142025.0611.3
llama-3.1-8b-SLRPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.05, Cost ($)=0.142025.0610.5
llama-3.1-8b-SLRPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.05, Cost ($)=0.142025.068.2
Llama-3.3-70B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=0.93, Cost ($)=0.932025.067.5
Llama-3.3-70B-ITPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.49, Cost ($)=0.822025.067.3
Llama-3.3-70B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=0.93, Cost ($)=0.932025.067
Llama-3.3-70B-ITPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.49, Cost ($)=0.822025.066.8
Llama-3.1-8BPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.20, Cost ($)=0.142025.066
Llama-3.1-8B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=1.37, Cost ($)=0.182025.066
Llama-3.3-70B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=0.93, Cost ($)=0.932025.065.9
Llama-3.3-70B-ITPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.49, Cost ($)=0.822025.065.6
Llama-3.1-8BPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.20, Cost ($)=0.142025.065.5
Llama-3.1-8B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=1.37, Cost ($)=0.182025.065.3
Llama-3.2-3B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=1.02, Cost ($)=0.132025.064.5
Llama-3.1-8BPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.20, Cost ($)=0.142025.063.8
Llama-3.2-3B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=1.02, Cost ($)=0.132025.063.8
Llama-3.2-3B-ITPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.10, Cost ($)=0.112025.063.6
Llama-3.1-8B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=1.37, Cost ($)=0.182025.063.4
Llama-3.2-3B-ITPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.10, Cost ($)=0.112025.062.6
Llama-3.2-3B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=1.02, Cost ($)=0.132025.061.8
Llama-3.2-3B-ITPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.10, Cost ($)=0.112025.061
Llama-3.2-1B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=1.17, Cost ($)=0.172025.060
Llama-3.2-1B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=1.17, Cost ($)=0.172025.060
Llama-3.2-1B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=1.17, Cost ($)=0.172025.060
Llama-3.2-1B-ITPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.47, Cost ($)=0.122025.060
Llama-3.2-1B-ITPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.47, Cost ($)=0.122025.060
Llama-3.2-1B-ITPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.47, Cost ($)=0.122025.060