List your company
ResearchFormerly WizWand

Symbolic Reasoning on Last Letter

0.819AccuracyIn-Writing-BF
0.415480.520240.6250.72976Jan 12, 2026Feb 1, 2026Feb 21, 2026Mar 13, 2026Apr 2, 2026Apr 22, 2026May 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
In-Writing-BFBackbone=LLaMA3-8B, Shot=12026.010.819
In-Writing-BFBackbone=LLaMA3-8B, Shot=42026.010.803
In-Writing-IFBackbone=LLaMA3-8B, Shot=42026.010.792
In-Writing-BFBackbone=Qwen3-8B, Shot=42026.010.787
In-Writing-IFBackbone=LLaMA3-8B, Shot=12026.010.779
In-Writing-BaseBackbone=LLaMA3-8B, Shot=42026.010.774
In-Writing-IFBackbone=Qwen3-8B, Shot=42026.010.77
In-Writing-BaseBackbone=Qwen3-8B, Shot=42026.010.759
In-Writing-BFBackbone=Qwen3-8B, Shot=12026.010.749
In-Writing-BaseBackbone=Qwen3-8B, Shot=02026.010.747
In-Writing-BFBackbone=Qwen3-8B, Shot=02026.010.74
In-Writing-IFBackbone=Qwen3-8B, Shot=12026.010.733
In-Writing-IFBackbone=Qwen3-8B, Shot=02026.010.731
ZS-CoT (teacher)Model=GPT-4o-mini, #Params=-2026.050.724
In-Writing-BaseBackbone=Qwen3-8B, Shot=12026.010.708
Vanilla-KDModel=FlanT5-large, #Params=780M2026.050.708
In-Writing-BFBackbone=LLaMA3-8B, Shot=02026.010.704
In-Writing-IFBackbone=LLaMA3-8B, Shot=02026.010.702
NL-to FormatBackbone=LLaMA3-8B, Shot=02026.010.701
In-Writing-BaseBackbone=LLaMA3-8B, Shot=02026.010.691
GateKDModel=FlanT5-base, #Params=250M2026.050.679
NL-to FormatBackbone=LLaMA3-8B, Shot=42026.010.676
Mentor-KDModel=FlanT5-base, #Params=250M2026.050.638
GateKDModel=FlanT5-small, #Params=80M2026.050.625
MCC-KDModel=FlanT5-base, #Params=250M2026.050.603
Vanilla-KDModel=FlanT5-base, #Params=250M2026.050.589
Mentor-KDModel=FlanT5-small, #Params=80M2026.050.581
MCC-KDModel=FlanT5-small, #Params=80M2026.050.54
Vanilla-KDModel=FlanT5-small, #Params=80M2026.050.524
NL-to FormatBackbone=LLaMA3-8B, Shot=12026.010.446
In-Writing-BaseBackbone=LLaMA3-8B, Shot=12026.010.431