Logical Reasoning on StrategyQA
89AccuracyFull
Evaluation Results
| Method | Links | |
|---|---|---|
| FullBackbone=Nemotron-Nano-8B2025.12 | 89 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 85 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 85 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 84 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 84 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 84 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 83 | |
| SnapKVBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 83 | |
| 8-shot CoTModel=PaLM-540B, Setting=Prev, # Paths=402023.05 | 81.6 | |
| 5-shot CoTModel=PaLM-540B, Setting=5-shot, # Paths=142023.05 | 81.4 | |
| H2OBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 76 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 76 | |
| Auto-CoTModel=PaLM-540B, Setting=0-shot, # Paths=142023.05 | 75.7 | |
| COSPModel=PaLM-540B, Setting=0-shot, # Paths=7+72023.05 | 75.2 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 74 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 72 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 71 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=5122025.12 | 69 | |
| Gt-MarginOrder score=Ours, Backbone=LLaDA-8B2026.02 | 68.5 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 68 | |
| 5-shot CoTModel=PaLM-62B, Setting=5-shot, # Paths=142023.05 | 67.9 | |
| FullBackbone=Deepseek-R1-Distill-Qwen-7B2025.12 | 67 | |
| 0-shot CoTModel=PaLM-540B, Setting=0-shot, # Paths=142023.05 | 66 | |
| MarginOrder score=Margin, Backbone=LLaDA-8B2026.02 | 65.5 | |
| ShadowKVBackbone=Nemotron-Nano-8B2025.12 | 65 | |
| COSPModel=PaLM-62B, Setting=0-shot, # Paths=7+72023.05 | 64.7 | |
| Gt-MarginOrder score=Ours w/o partial plan, partial plan=false, Backbone=LLaDA-8B2026.02 | 64.5 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 64 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 64 | |
| Gt-MarginOrder score=Ours w/o token pred, token prediction hypothesis=false, Backbone=LLaDA-8B2026.02 | 63.5 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 63 | |
| ShadowKVBackbone=Deepseek-R1-Distill-Qwen-7B2025.12 | 60 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 60 | |
| Auto-CoTModel=PaLM-62B, Setting=0-shot, # Paths=142023.05 | 59.4 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 59 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 59 | |
| 0-shot CoTModel=PaLM-62B, Setting=0-shot, # Paths=142023.05 | 57.2 | |
| SnapKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 57 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 55 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=3842025.12 | 52 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 45 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 44 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 42 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=5122025.12 | 42 | |
| RKVBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 42 | |
| RandomOrder score=Random, Backbone=LLaDA-8B2026.02 | 41.5 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=2562025.12 | 39 | |
| KnormBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 38 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 34 | |
| H2OBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 33 | |
| StreamingLLMBackbone=Nemotron-Nano-8B, Budget=1282025.12 | 24 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=3842025.12 | 22 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 14 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 12 | |
| RKVBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 5 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=2562025.12 | 5 | |
| KnormBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 0 | |
| StreamingLLMBackbone=Deepseek-R1-Distill-Qwen-7B, Budget=1282025.12 | 0 |