Detective Puzzle Reasoning on DP Complex
79.3SASABA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SABANormalized inference cost (T)=9.22026.04 | 79.3 | 73.4 | 72.7 | 83.3 | |
| GoTNormalized inference cost (T)=35.72026.04 | 69.8 | 69.8 | 66.9 | 77.3 | |
| SELF-DISC.Normalized inference cost (T)=5.02026.04 | 68.8 | 64.2 | 65.3 | 75.6 | |
| SABABackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=9.82026.04 | 68.8 | 72.1 | 62.4 | 75.7 | |
| S^2RNormalized inference cost (T)=18.42026.04 | 68.4 | 63.7 | 67.2 | 77.1 | |
| S^2RBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=18.52026.04 | 67.5 | 61.8 | 55.7 | 64.9 | |
| CRITICNormalized inference cost (T)=29.32026.04 | 66.4 | 64.6 | 63.5 | 73.6 | |
| SC(k=5)Normalized inference cost (T)=12.0, k=52026.04 | 62.7 | 62.1 | 62.3 | 71.3 | |
| GoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=35.52026.04 | 61 | 62.1 | 58.5 | 65.1 | |
| SELF-DISC.Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=4.82026.04 | 56.7 | 58.5 | 51.8 | 62.8 | |
| Self-RefineNormalized inference cost (T)=6.32026.04 | 55.6 | 62 | 61.4 | 65.5 | |
| CRITICBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=29.22026.04 | 49.9 | 55.1 | 48.5 | 60.4 | |
| CoTNormalized inference cost (T)=2.52026.04 | 45.4 | 61 | 59.7 | 61.9 | |
| SC(k=5)Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=12.12026.04 | 42.5 | 42.7 | 42.4 | 45.2 | |
| DirectNormalized inference cost (T)=1.02026.04 | 40.7 | 59.4 | 58.9 | 58.7 | |
| CoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=2.42026.04 | 35.7 | 44.8 | 40.7 | 35.5 | |
| Self-RefineBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=6.22026.04 | 32.7 | 45.4 | 43.2 | 38.2 | |
| DirectBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=1.02026.04 | 32.3 | 42.4 | 38 | 32.4 |