List your company
ResearchFormerly WizWand

Logical Reasoning on StrategyQA

89AccuracyFull
-3.5620.4744.568.53May 23, 2023Nov 4, 2023Apr 18, 2024Oct 1, 2024Mar 15, 2025Aug 28, 2025Feb 10, 2026
Updated 8mo ago

Evaluation results

MethodLinks
FullBackbone=Nemotron-Nano-8B2025.1289
H2OBackbone=Nemotron-Nano-8B, Budget=3842025.1285
SnapKVBackbone=Nemotron-Nano-8B, Budget=2562025.1285
H2OBackbone=Nemotron-Nano-8B, Budget=2562025.1284
SnapKVBackbone=Nemotron-Nano-8B, Budget=3842025.1284
SnapKVBackbone=Nemotron-Nano-8B, Budget=5122025.1284
H2OBackbone=Nemotron-Nano-8B, Budget=5122025.1283
SnapKVBackbone=Nemotron-Nano-8B, Budget=1282025.1283
8-shot CoTModel=PaLM-540B, Setting=Prev, # Paths=402023.0581.6
5-shot CoTModel=PaLM-540B, Setting=5-shot, # Paths=142023.0581.4
H2OBackbone=Nemotron-Nano-8B, Budget=1282025.1276
KnormBackbone=Nemotron-Nano-8B, Budget=5122025.1276
Auto-CoTModel=PaLM-540B, Setting=0-shot, # Paths=142023.0575.7
COSPModel=PaLM-540B, Setting=0-shot, # Paths=7+72023.0575.2
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.1274
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.1272
RKVBackbone=Nemotron-Nano-8B, Budget=5122025.1271
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=5122025.1269
Gt-MarginOrder score=Ours, Backbone=LLaDA-8B2026.0268.5
KnormBackbone=Nemotron-Nano-8B, Budget=3842025.1268
5-shot CoTModel=PaLM-62B, Setting=5-shot, # Paths=142023.0567.9
FullBackbone=Deepseek-R1-Distill-Qwen-7B2025.1267
0-shot CoTModel=PaLM-540B, Setting=0-shot, # Paths=142023.0566
MarginOrder score=Margin, Backbone=LLaDA-8B2026.0265.5
ShadowKVBackbone=Nemotron-Nano-8B2025.1265
COSPModel=PaLM-62B, Setting=0-shot, # Paths=7+72023.0564.7
Gt-MarginOrder score=Ours w/o partial plan, partial plan=false, Backbone=LLaDA-8B2026.0264.5
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.1264
RKVBackbone=Nemotron-Nano-8B, Budget=3842025.1264
Gt-MarginOrder score=Ours w/o token pred, token prediction hypothesis=false, Backbone=LLaDA-8B2026.0263.5
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.1263
ShadowKVBackbone=Deepseek-R1-Distill-Qwen-7B2025.1260
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.1260
Auto-CoTModel=PaLM-62B, Setting=0-shot, # Paths=142023.0559.4
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.1259
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.1259
0-shot CoTModel=PaLM-62B, Setting=0-shot, # Paths=142023.0557.2
SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.1257
KnormBackbone=Nemotron-Nano-8B, Budget=2562025.1255
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=3842025.1252
RKVBackbone=Nemotron-Nano-8B, Budget=2562025.1245
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.1244
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.1242
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.1242
RKVBackbone=Nemotron-Nano-8B, Budget=1282025.1242
RandomOrder score=Random, Backbone=LLaDA-8B2026.0241.5
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=2562025.1239
KnormBackbone=Nemotron-Nano-8B, Budget=1282025.1238
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.1234
H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.1233
StreamingLLMBackbone=Nemotron-Nano-8B, Budget=1282025.1224
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.1222
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.1214
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.1212
RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.125
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.125
KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.120
StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.120