Detective Puzzle Reasoning on DP Easy
85.7SA ScoreSABA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SABANormalized inference cost (T)=9.22026.04 | 85.7 | 84.3 | 85.8 | 93.8 | |
| GoTNormalized inference cost (T)=35.72026.04 | 84.1 | 78.6 | 82.2 | 90.5 | |
| SELF-DISC.Normalized inference cost (T)=5.02026.04 | 81 | 72.3 | 78.5 | 87.1 | |
| S^2RNormalized inference cost (T)=18.42026.04 | 79.3 | 70.8 | 76.3 | 89.7 | |
| CRITICNormalized inference cost (T)=29.32026.04 | 77.5 | 67.9 | 70.8 | 88.6 | |
| SABABackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=9.82026.04 | 74.3 | 65 | 72.7 | 85.4 | |
| SC(k=5)Normalized inference cost (T)=12.0, k=52026.04 | 72 | 68.8 | 69.2 | 81.5 | |
| Self-RefineNormalized inference cost (T)=6.32026.04 | 70.9 | 66.9 | 68.5 | 80.3 | |
| CoTNormalized inference cost (T)=2.52026.04 | 68.1 | 77.5 | 65.1 | 75.9 | |
| GoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=35.52026.04 | 67.1 | 58.9 | 70.6 | 78.1 | |
| S^2RBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=18.52026.04 | 65.7 | 58 | 67.5 | 74.8 | |
| DirectNormalized inference cost (T)=1.02026.04 | 65.2 | 65 | 60.5 | 71 | |
| SELF-DISC.Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=4.82026.04 | 63 | 56.7 | 69 | 74.5 | |
| CRITICBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=29.22026.04 | 58 | 52.4 | 65 | 72.4 | |
| SC(k=5)Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=12.12026.04 | 51.9 | 48.3 | 50.6 | 65.1 | |
| CoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=2.42026.04 | 42.8 | 52 | 51.3 | 49.5 | |
| Self-RefineBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=6.22026.04 | 41.8 | 50.5 | 50.1 | 52 | |
| DirectBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=1.02026.04 | 40.1 | 48.2 | 47.1 | 45.3 |