Detective Puzzle Reasoning on DP Medium
83.2SASABA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SABANormalized inference cost (T)=9.22026.04 | 83.2 | 74.9 | 73.5 | 92.2 | |
| GoTNormalized inference cost (T)=35.72026.04 | 77 | 79.9 | 72.1 | 87.9 | |
| S^2RNormalized inference cost (T)=18.42026.04 | 76.3 | 66.5 | 72.2 | 88 | |
| SELF-DISC.Normalized inference cost (T)=5.02026.04 | 75.1 | 68.4 | 69.9 | 86.9 | |
| CRITICNormalized inference cost (T)=29.32026.04 | 73.6 | 65.6 | 68.5 | 87.6 | |
| SABABackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=9.82026.04 | 72.2 | 68.8 | 64.9 | 82.9 | |
| SC(k=5)Normalized inference cost (T)=12.0, k=52026.04 | 70.3 | 64.6 | 66.4 | 79.9 | |
| Self-RefineNormalized inference cost (T)=6.32026.04 | 65.5 | 66.7 | 65 | 76.4 | |
| GoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=35.52026.04 | 64 | 56.4 | 62.1 | 75.5 | |
| S^2RBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=18.52026.04 | 61.8 | 55.7 | 59.8 | 72.7 | |
| SELF-DISC.Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=4.82026.04 | 59.9 | 51.8 | 63.1 | 69.8 | |
| CoTNormalized inference cost (T)=2.52026.04 | 58.6 | 64.9 | 62 | 72.2 | |
| CRITICBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=29.22026.04 | 52.5 | 48.2 | 52.3 | 67.9 | |
| SC(k=5)Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=12.12026.04 | 51.2 | 45.1 | 45 | 58.7 | |
| DirectNormalized inference cost (T)=1.02026.04 | 48.1 | 64.3 | 58 | 62.5 | |
| Self-RefineBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=6.22026.04 | 38.5 | 48.2 | 44 | 42.4 | |
| CoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=2.42026.04 | 38 | 45.9 | 48 | 48.1 | |
| DirectBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=1.02026.04 | 35.6 | 44.5 | 45.3 | 35.1 |