Logical Reasoning on SLR-BENCH (test)
11.3LRLllama-3.1-8b-SLR
Evaluation Results
| Method | Links | |
|---|---|---|
| llama-3.1-8b-SLRPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.05, Cost ($)=0.142025.06 | 11.3 | |
| llama-3.1-8b-SLRPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.05, Cost ($)=0.142025.06 | 10.5 | |
| llama-3.1-8b-SLRPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.05, Cost ($)=0.142025.06 | 8.2 | |
| Llama-3.3-70B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=0.93, Cost ($)=0.932025.06 | 7.5 | |
| Llama-3.3-70B-ITPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.49, Cost ($)=0.822025.06 | 7.3 | |
| Llama-3.3-70B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=0.93, Cost ($)=0.932025.06 | 7 | |
| Llama-3.3-70B-ITPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.49, Cost ($)=0.822025.06 | 6.8 | |
| Llama-3.1-8BPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.20, Cost ($)=0.142025.06 | 6 | |
| Llama-3.1-8B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=1.37, Cost ($)=0.182025.06 | 6 | |
| Llama-3.3-70B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=0.93, Cost ($)=0.932025.06 | 5.9 | |
| Llama-3.3-70B-ITPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.49, Cost ($)=0.822025.06 | 5.6 | |
| Llama-3.1-8BPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.20, Cost ($)=0.142025.06 | 5.5 | |
| Llama-3.1-8B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=1.37, Cost ($)=0.182025.06 | 5.3 | |
| Llama-3.2-3B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=1.02, Cost ($)=0.132025.06 | 4.5 | |
| Llama-3.1-8BPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.20, Cost ($)=0.142025.06 | 3.8 | |
| Llama-3.2-3B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=1.02, Cost ($)=0.132025.06 | 3.8 | |
| Llama-3.2-3B-ITPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.10, Cost ($)=0.112025.06 | 3.6 | |
| Llama-3.1-8B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=1.37, Cost ($)=0.182025.06 | 3.4 | |
| Llama-3.2-3B-ITPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.10, Cost ($)=0.112025.06 | 2.6 | |
| Llama-3.2-3B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=1.02, Cost ($)=0.132025.06 | 1.8 | |
| Llama-3.2-3B-ITPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.10, Cost ($)=0.112025.06 | 1 | |
| Llama-3.2-1B-ITPass@k=1, Chain-of-Thought (CoT)=true, Tokens (M)=1.17, Cost ($)=0.172025.06 | 0 | |
| Llama-3.2-1B-ITPass@k=4, Chain-of-Thought (CoT)=true, Tokens (M)=1.17, Cost ($)=0.172025.06 | 0 | |
| Llama-3.2-1B-ITPass@k=8, Chain-of-Thought (CoT)=true, Tokens (M)=1.17, Cost ($)=0.172025.06 | 0 | |
| Llama-3.2-1B-ITPass@k=1, Chain-of-Thought (CoT)=false, Tokens (M)=0.47, Cost ($)=0.122025.06 | 0 | |
| Llama-3.2-1B-ITPass@k=4, Chain-of-Thought (CoT)=false, Tokens (M)=0.47, Cost ($)=0.122025.06 | 0 | |
| Llama-3.2-1B-ITPass@k=8, Chain-of-Thought (CoT)=false, Tokens (M)=0.47, Cost ($)=0.122025.06 | 0 |