Symbolic Reasoning on Last Letter
0.819AccuracyIn-Writing-BF
Evaluation Results
| Method | Links | |
|---|---|---|
| In-Writing-BFBackbone=LLaMA3-8B, Shot=12026.01 | 0.819 | |
| In-Writing-BFBackbone=LLaMA3-8B, Shot=42026.01 | 0.803 | |
| In-Writing-IFBackbone=LLaMA3-8B, Shot=42026.01 | 0.792 | |
| In-Writing-BFBackbone=Qwen3-8B, Shot=42026.01 | 0.787 | |
| In-Writing-IFBackbone=LLaMA3-8B, Shot=12026.01 | 0.779 | |
| In-Writing-BaseBackbone=LLaMA3-8B, Shot=42026.01 | 0.774 | |
| In-Writing-IFBackbone=Qwen3-8B, Shot=42026.01 | 0.77 | |
| In-Writing-BaseBackbone=Qwen3-8B, Shot=42026.01 | 0.759 | |
| In-Writing-BFBackbone=Qwen3-8B, Shot=12026.01 | 0.749 | |
| In-Writing-BaseBackbone=Qwen3-8B, Shot=02026.01 | 0.747 | |
| In-Writing-BFBackbone=Qwen3-8B, Shot=02026.01 | 0.74 | |
| In-Writing-IFBackbone=Qwen3-8B, Shot=12026.01 | 0.733 | |
| In-Writing-IFBackbone=Qwen3-8B, Shot=02026.01 | 0.731 | |
| ZS-CoT (teacher)Model=GPT-4o-mini, #Params=-2026.05 | 0.724 | |
| In-Writing-BaseBackbone=Qwen3-8B, Shot=12026.01 | 0.708 | |
| Vanilla-KDModel=FlanT5-large, #Params=780M2026.05 | 0.708 | |
| In-Writing-BFBackbone=LLaMA3-8B, Shot=02026.01 | 0.704 | |
| In-Writing-IFBackbone=LLaMA3-8B, Shot=02026.01 | 0.702 | |
| NL-to FormatBackbone=LLaMA3-8B, Shot=02026.01 | 0.701 | |
| In-Writing-BaseBackbone=LLaMA3-8B, Shot=02026.01 | 0.691 | |
| GateKDModel=FlanT5-base, #Params=250M2026.05 | 0.679 | |
| NL-to FormatBackbone=LLaMA3-8B, Shot=42026.01 | 0.676 | |
| Mentor-KDModel=FlanT5-base, #Params=250M2026.05 | 0.638 | |
| GateKDModel=FlanT5-small, #Params=80M2026.05 | 0.625 | |
| MCC-KDModel=FlanT5-base, #Params=250M2026.05 | 0.603 | |
| Vanilla-KDModel=FlanT5-base, #Params=250M2026.05 | 0.589 | |
| Mentor-KDModel=FlanT5-small, #Params=80M2026.05 | 0.581 | |
| MCC-KDModel=FlanT5-small, #Params=80M2026.05 | 0.54 | |
| Vanilla-KDModel=FlanT5-small, #Params=80M2026.05 | 0.524 | |
| NL-to FormatBackbone=LLaMA3-8B, Shot=12026.01 | 0.446 | |
| In-Writing-BaseBackbone=LLaMA3-8B, Shot=12026.01 | 0.431 |