Logical Reasoning on Countdown
63.2AccuracyFlowTracer
Evaluation Results
| Method | Links | |
|---|---|---|
| FlowTracerBackbone=Qwen3-4B-Base, Context window=1K2026.06 | 63.2 | |
| AttentionBackbone=Qwen3-4B-Base, Context window=1K2026.06 | 60.4 | |
| High-entropyBackbone=Qwen3-4B-Base, Context window=1K2026.06 | 57.7 | |
| RandomBackbone=Qwen3-4B-Base, Context window=1K2026.06 | 55 | |
| GRPOBackbone=Qwen3-4B-Base, Context window=1K2026.06 | 52.6 | |
| + EOS Temp.Sampling Method=+ EOS Temp., Timestep Budget (T)=1282026.04 | 52 | |
| OracleBackbone=Dream-7B-Base, Shots=4-shot2026.04 | 51.3 | |
| + EOS Temp.Sampling Method=+ EOS Temp., Timestep Budget (T)=642026.04 | 51.2 | |
| OracleBackbone=LLaDA-8B-Base, Shots=4-shot2026.04 | 49.6 | |
| Auto-ICLBackbone=Dream-7B-Base, Shots=4-shot, Latency (s)=0.702026.04 | 49.5 | |
| BothSampling Method=Both, Timestep Budget (T)=1282026.04 | 48.4 | |
| VanillaBackbone=Dream-7B-Base, Shots=4-shot, Latency (s)=0.712026.04 | 48.1 | |
| + PlannerSampling Method=+ Planner, Timestep Budget (T)=642026.04 | 46.9 | |
| BothSampling Method=Both, Timestep Budget (T)=642026.04 | 46.5 | |
| Prob MarginSampling Method=Prob Margin, Timestep Budget (T)=642026.04 | 46.1 | |
| + PlannerSampling Method=+ Planner, Timestep Budget (T)=1282026.04 | 45.7 | |
| Prob MarginSampling Method=Prob Margin, Timestep Budget (T)=1282026.04 | 44.5 | |
| TemperatureSampling Method=Temperature, Timestep Budget (T)=1282026.04 | 43.8 | |
| Auto-ICLBackbone=LLaDA-8B-Base, Shots=4-shot, Latency (s)=0.742026.04 | 42.2 | |
| Top1 ProbSampling Method=Top1 Prob, Timestep Budget (T)=1282026.04 | 41.8 | |
| Top1 ProbSampling Method=Top1 Prob, Timestep Budget (T)=642026.04 | 41.4 | |
| TemperatureSampling Method=Temperature, Timestep Budget (T)=642026.04 | 41.4 | |
| Init. PositionSampling Method=Init. Position, Timestep Budget (T)=1282026.04 | 39.5 | |
| Init. PositionSampling Method=Init. Position, Timestep Budget (T)=642026.04 | 36.7 | |
| PrefixBackbone=Dream-7B-Base, Shots=4-shot, Latency (s)=0.712026.04 | 34.7 | |
| PrefixBackbone=LLaDA-8B-Base, Shots=4-shot, Latency (s)=0.752026.04 | 32 | |
| VanillaBackbone=LLaDA-8B-Base, Shots=4-shot, Latency (s)=0.742026.04 | 31 | |
| AncestralSampling Method=Ancestral, Timestep Budget (T)=642026.04 | 25.4 | |
| AncestralSampling Method=Ancestral, Timestep Budget (T)=1282026.04 | 23.8 | |
| RandomBackbone=Dream-7B-Base, Shots=4-shot, Latency (s)=0.722026.04 | 21.8 | |
| RandomBackbone=LLaDA-8B-Base, Shots=4-shot, Latency (s)=0.742026.04 | 17 |