Logical Reasoning on Shuffled Objects
92.1AccuracyGateKD
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| GateKDModel=FlanT5-base, #Params=250M2026.05 | 92.1 | — | — | |
| GateKDModel=T5-base, #Params=250M2026.05 | 90.6 | — | — | |
| In-Writing-BFBackbone=Qwen3-8B, Shot=02026.01 | 89.9 | — | — | |
| Vanilla-KDModel=FlanT5-large, #Params=780M2026.05 | 89.7 | — | — | |
| In-Writing-IFBackbone=Qwen3-8B, Shot=02026.01 | 88.7 | — | — | |
| In-Writing-BFBackbone=Qwen3-8B, Shot=42026.01 | 88.7 | — | — | |
| In-Writing-BFBackbone=Qwen3-8B, Shot=12026.01 | 88.3 | — | — | |
| Mentor-KDModel=FlanT5-base, #Params=250M2026.05 | 88.3 | — | — | |
| Vanilla-KDModel=T5-large, #Params=780M2026.05 | 88.1 | — | — | |
| In-Writing-BaseBackbone=Qwen3-8B, Shot=02026.01 | 87.6 | — | — | |
| In-Writing-IFBackbone=Qwen3-8B, Shot=42026.01 | 86.7 | — | — | |
| MCC-KDModel=FlanT5-base, #Params=250M2026.05 | 85.1 | — | — | |
| Mentor-KDModel=T5-base, #Params=250M2026.05 | 84.9 | — | — | |
| In-Writing-IFBackbone=Qwen3-8B, Shot=12026.01 | 84.8 | — | — | |
| ZS-CoT (teacher)Model=GPT-4o-mini, #Params=-2026.05 | 84.3 | — | — | |
| In-Writing-BaseBackbone=Qwen3-8B, Shot=42026.01 | 84 | — | — | |
| GateKDModel=FlanT5-small, #Params=80M2026.05 | 83.7 | — | — | |
| ZS-CoT (teacher)Model=GPT-4o-mini, #Params=-2026.05 | 83.1 | — | — | |
| In-Writing-BaseBackbone=Qwen3-8B, Shot=12026.01 | 82.6 | — | — | |
| Vanilla-KDModel=FlanT5-base, #Params=250M2026.05 | 82.6 | — | — | |
| MCC-KDModel=T5-base, #Params=250M2026.05 | 81 | — | — | |
| GateKDModel=T5-small, #Params=80M2026.05 | 80.8 | — | — | |
| Vanilla-KDModel=T5-base, #Params=250M2026.05 | 78.4 | — | — | |
| Mentor-KDModel=FlanT5-small, #Params=80M2026.05 | 76.4 | — | — | |
| Mentor-KDModel=T5-small, #Params=80M2026.05 | 72.9 | — | — | |
| MCC-KDModel=FlanT5-small, #Params=80M2026.05 | 70 | — | — | |
| Vanilla-KDModel=FlanT5-small, #Params=80M2026.05 | 68.2 | — | — | |
| MCC-KDModel=T5-small, #Params=80M2026.05 | 66.1 | — | — | |
| Vanilla-KDModel=T5-small, #Params=80M2026.05 | 63.7 | — | — | |
| In-Writing-BFBackbone=LLaMA3-8B, Shot=12026.01 | 48.3 | — | — | |
| In-Writing-IFBackbone=LLaMA3-8B, Shot=42026.01 | 46.7 | — | — | |
| In-Writing-BFBackbone=LLaMA3-8B, Shot=42026.01 | 46.4 | — | — | |
| In-Writing-IFBackbone=LLaMA3-8B, Shot=12026.01 | 45.8 | — | — | |
| In-Writing-BaseBackbone=LLaMA3-8B, Shot=02026.01 | 39 | — | — | |
| In-Writing-BaseBackbone=LLaMA3-8B, Shot=42026.01 | 38.2 | — | — | |
| In-Writing-IFBackbone=LLaMA3-8B, Shot=02026.01 | 34.8 | — | — | |
| NL-to FormatBackbone=LLaMA3-8B, Shot=02026.01 | 27 | — | — | |
| In-Writing-BFBackbone=LLaMA3-8B, Shot=02026.01 | 24.3 | — | — | |
| In-Writing-BaseBackbone=LLaMA3-8B, Shot=12026.01 | 22.4 | — | — | |
| zero-shotReasoning Method=Zero-shot2022.05 | — | 31.3 | 29.7 | |
| zero-shot-cotReasoning Method=Zero-shot-CoT2022.05 | — | 52.4 | 52.9 |