Logical Reasoning on K&K tasks
2.13Perplexity (3ppl)Vanilla
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| VanillaModel=Qwen3-1.7B-Base2026.05 | 2.13 | 1.5 | 0.37 | 0.75 | 0.12 | 0 | 0.81 | |
| VanillaModel=Qwen3-4B-Base2026.05 | 9.25 | 7.63 | 4.63 | 3 | 1.75 | 0.63 | 4.48 | |
| TTRLModel=Qwen3-1.7B-Base2026.05 | 10.63 | 4.87 | 1.88 | 1.75 | 1.25 | 1.25 | 3.6 | |
| ProbModel=Qwen3-1.7B-Base2026.05 | 11.63 | 2.88 | 5.12 | 2.75 | 0.63 | 0 | 3.6 | |
| EntropyModel=Qwen3-1.7B-Base2026.05 | 13.12 | 2.88 | 5.12 | 2.75 | 0.63 | 0 | 4.08 | |
| RandomModel=Qwen3-1.7B-Base2026.05 | 18.25 | 10 | 5 | 4.25 | 0.88 | 0 | 6.4 | |
| VanillaModel=Phi4-mini-instruct2026.05 | 21.88 | 16 | 8 | 6.37 | 3 | 1.75 | 9.5 | |
| TTRLModel=Qwen3-8B-Base2026.05 | 24.88 | 10.37 | 6.75 | 8.12 | 2 | 4 | 9.35 | |
| VanillaModel=Qwen3-8B-Base2026.05 | 25.25 | 17.5 | 13.88 | 8.75 | 5.5 | 3.62 | 12.42 | |
| RandomModel=Qwen3-8B-Base2026.05 | 25.75 | — | 8.25 | 10.63 | 2.37 | 4.13 | 10.54 | |
| OracleModel=Qwen3-1.7B-Base2026.05 | 27.38 | 20.87 | 15.75 | 15.25 | 4 | 5 | 14.71 | |
| EntropyModel=Qwen3-4B-Base2026.05 | 28.25 | 16.75 | 14.12 | 10.5 | 3 | 4.13 | 12.79 | |
| ProbModel=Qwen3-4B-Base2026.05 | 28.37 | 18 | 11.87 | 11.12 | 2.62 | 4.87 | 12.81 | |
| GTModel=Qwen3-1.7B-Base2026.05 | 40.25 | 30.13 | 19 | 20.87 | 6.75 | 8.12 | 20.85 | |
| ProbModel=Qwen3-8B-Base2026.05 | 41.12 | 30.38 | 11.2 | 14.25 | 5 | 4 | 17.79 | |
| CAREModel=Qwen3-1.7B-Base2026.05 | 41.37 | 30.5 | 17.75 | 18.63 | 6.5 | 11.34 | 21.02 | |
| RandomModel=Phi4-mini-instruct2026.05 | 45.5 | 31.87 | 19 | 18.5 | 13.12 | 9.13 | 22.85 | |
| EntropyModel=Qwen3-8B-Base2026.05 | 46 | 41 | 32.62 | 32 | 14 | 13 | 29.77 | |
| TTRLModel=Phi4-mini-instruct2026.05 | 46.88 | 33.63 | 21.63 | 20.37 | 13.5 | 10.13 | 24.35 | |
| ProbModel=Phi4-mini-instruct2026.05 | 47.37 | 33.88 | 24.25 | 20.5 | 13.38 | 12.25 | 25.27 | |
| EntropyModel=Phi4-mini-instruct2026.05 | 48.25 | 35 | 26.25 | 23.25 | 14.25 | 10.88 | 26.31 | |
| TTRLModel=Qwen3-4B-Base2026.05 | 50.25 | 35.37 | 28.13 | 21.88 | 12.87 | 9.25 | 26.29 | |
| OracleModel=Phi4-mini-instruct2026.05 | 54.25 | 44.5 | 32.88 | 33.5 | 23.5 | 17.25 | 34.31 | |
| RandomModel=Qwen3-4B-Base2026.05 | 56.63 | 43 | 28.75 | 24.38 | 13.88 | 10.88 | 29.58 | |
| CAREModel=Phi4-mini-instruct2026.05 | 64.5 | 54.62 | — | 36 | 25.5 | 22.25 | 40.04 | |
| GTModel=Phi4-mini-instruct2026.05 | 64.75 | 54.12 | 41.75 | 38.12 | 27.88 | 21 | 41.27 | |
| OracleModel=Qwen3-4B-Base2026.05 | 89 | 87.25 | 74.87 | 63.38 | 59.25 | 54.37 | 71.42 | |
| OracleModel=Qwen3-8B-Base2026.05 | 95 | 93.5 | 86.75 | 78.5 | 74.25 | 69.63 | 82.94 | |
| CAREModel=Qwen3-4B-Base2026.05 | 97.75 | 96.25 | 89.5 | 83.13 | 76.75 | 68.62 | 85.33 | |
| GTModel=Qwen3-8B-Base2026.05 | 99 | 97.87 | 97 | 89 | 88.75 | 84.62 | 92.71 | |
| GTModel=Qwen3-4B-Base2026.05 | 99.37 | 98.12 | 95.25 | 88.38 | 85.12 | 74.75 | 90.17 | |
| CAREModel=Qwen3-8B-Base2026.05 | 99.37 | 98 | 93.25 | 91 | 86.75 | 77.38 | 90.96 |