Mathematical Reasoning on GSM8K Hard
61.8AccuracyTALE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| TALEBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=22025.10 | 61.8 | — | — | — | 28 | -43.6 | |
| BaselineBackbone=LLaMA 3.1 8B, Shots=8-shot2025.10 | 60.67 | — | — | — | — | — | |
| Best ModelBackbone=LLaMA 3.1 8B, Shots=8-shot, #D=02025.10 | 60.67 | — | — | 1 | — | — | |
| BSBABackbone=LLaMA 3.1 8B, Shots=8-shot, #D=02025.10 | 60.67 | — | — | 1 | — | — | |
| TALEBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=12025.10 | 59 | — | — | — | 30 | -2.9 | |
| BSBABackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=52025.10 | 43.99 | — | — | — | — | -17.6 | |
| BaselineBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 43.8 | — | — | — | — | — | |
| BSBABackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=42025.10 | 39.4 | — | — | — | — | -11.7 | |
| BaselineBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 39 | — | — | — | — | — | |
| TALEBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=12025.10 | 39 | — | — | — | 28 | -3.1 | |
| BSBABackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=32025.10 | 37 | — | — | — | — | -9.1 | |
| TALEBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=22025.10 | 33 | — | — | — | 24 | -6.2 | |
| BaselineBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 32 | — | — | — | — | — | |
| BSBABackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=42025.10 | 26.1 | — | — | — | — | -12.3 | |
| BaselineBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.10 | 24 | — | — | — | — | — | |
| FULL COTBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 21.6 | — | — | — | — | — | |
| FULL COTBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 20.5 | — | — | — | — | — | |
| KAVABackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 15.2 | — | — | — | — | — | |
| CODIBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 15 | — | — | — | — | — | |
| FULL COTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 14.8 | — | — | — | — | — | |
| KAVABackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 14.8 | — | — | — | — | — | |
| CODIBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 13.6 | — | — | — | — | — | |
| PCCOTBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 13.5 | — | — | — | — | — | |
| FULL COTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 13.3 | — | — | — | — | — | |
| PCCOTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 12.9 | — | — | — | — | — | |
| KAVABackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 12.8 | — | — | — | — | — | |
| KAVABackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 12.7 | — | — | — | — | — | |
| FULL COTBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 12.6 | — | — | — | — | — | |
| FULL COTBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 12.6 | — | — | — | — | — | |
| CODIBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 12.6 | — | — | — | — | — | |
| PCCOTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 12.3 | — | — | — | — | — | |
| CODIBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 11.5 | — | — | — | — | — | |
| CoTNumber of latent tokens=02026.02 | 11.3 | 0.11 | 100 | — | — | — | |
| PCCOTBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 11 | — | — | — | — | — | |
| KAVABackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 10.8 | — | — | — | — | — | |
| NO-COTBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 10.5 | — | — | — | — | — | |
| NO-COTBackbone=LLAMA3.2 - 3B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 10.5 | — | — | — | — | — | |
| KAVABackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 10.2 | — | — | — | — | — | |
| COCONUTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 9.9 | — | — | — | — | — | |
| CODIBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 8.1 | — | — | — | — | — | |
| NO-COTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 7.7 | — | — | — | — | — | |
| NO-COTBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 7.4 | — | — | — | — | — | |
| NO-COTBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 7.4 | — | — | — | — | — | |
| NO-COTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 7.4 | — | — | — | — | — | |
| COCONUTNumber of latent tokens=62026.02 | 6.9 | 0.63 | 10.9 | — | — | — | |
| CODIBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 4.9 | — | — | — | — | — | |
| OneLatentNumber of latent tokens=12026.02 | 4.58 | 0.9 | 5.1 | — | — | — | |
| ICOTBackbone=LLAMA3.2 - 1B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 4.4 | — | — | — | — | — | |
| iCoTNumber of latent tokens=02026.02 | 4.3 | 0.94 | 4.59 | — | — | — | |
| PCCOTBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG-NL2025.10 | 4.2 | — | — | — | — | — | |
| PCCOTBackbone=QWEN2.5 - 0.5B - INSTRUCT, Training Configuration=GSM8k-AUG2025.10 | 4.1 | — | — | — | — | — | |
| No CoTNumber of latent tokens=02026.02 | 1.36 | 0.29 | 4.63 | — | — | — | |
| SFT-CoTModel=LLaMA-3.1-8B, Explicit CoT=true2026.06 | 0.343 | — | — | — | — | — | |
| SFT-CoTModel=LLaMA-3.2-3B, Explicit CoT=true2026.06 | 0.325 | — | — | — | — | — | |
| LoRi-EModel=LLaMA-3.1-8B, Explicit CoT=false2026.06 | 0.272 | — | — | — | — | — | |
| LoRiModel=LLaMA-3.1-8B, Explicit CoT=false2026.06 | 0.261 | — | — | — | — | — | |
| LoRi-EModel=LLaMA-3.2-3B, Explicit CoT=false2026.06 | 0.241 | — | — | — | — | — | |
| LoRiModel=LLaMA-3.2-3B, Explicit CoT=false2026.06 | 0.219 | — | — | — | — | — | |
| SFT-CoTModel=LLaMA-3.2-1B, Explicit CoT=true2026.06 | 0.201 | — | — | — | — | — | |
| LoRi-EModel=Qwen2.5-0.5B, Explicit CoT=false2026.06 | 0.169 | — | — | — | — | — | |
| SIM-CoTModel=LLaMA-3.1-8B, Explicit CoT=false2026.06 | 0.163 | — | — | — | — | — | |
| SFT-CoTModel=Qwen2.5-0.5B, Explicit CoT=true2026.06 | 0.162 | — | — | — | — | — | |
| CODIModel=LLaMA-3.1-8B, Explicit CoT=false2026.06 | 0.155 | — | — | — | — | — | |
| LoRiModel=Qwen2.5-0.5B, Explicit CoT=false2026.06 | 0.152 | — | — | — | — | — | |
| KAVAModel=LLaMA-3.2-3B, Explicit CoT=false2026.06 | 0.152 | — | — | — | — | — | |
| CODIModel=LLaMA-3.2-3B, Explicit CoT=false2026.06 | 0.15 | — | — | — | — | — | |
| LoRi-EModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.149 | — | — | — | — | — | |
| SIM-CoTModel=LLaMA-3.2-3B, Explicit CoT=false2026.06 | 0.146 | — | — | — | — | — | |
| LoRiModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.139 | — | — | — | — | — | |
| PCCoTModel=LLaMA-3.2-3B, Explicit CoT=false2026.06 | 0.135 | — | — | — | — | — | |
| PCCoTModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.129 | — | — | — | — | — | |
| CODIModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.128 | — | — | — | — | — | |
| KAVAModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.127 | — | — | — | — | — | |
| KAVAModel=Qwen2.5-0.5B, Explicit CoT=false2026.06 | 0.108 | — | — | — | — | — | |
| NoCoTModel=LLaMA-3.2-3B, Explicit CoT=false2026.06 | 0.105 | — | — | — | — | — | |
| COCONUTModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.099 | — | — | — | — | — | |
| NoCoTModel=LLaMA-3.1-8B, Explicit CoT=false2026.06 | 0.098 | — | — | — | — | — | |
| CODIModel=Qwen2.5-0.5B, Explicit CoT=false2026.06 | 0.081 | — | — | — | — | — | |
| NoCoTModel=Qwen2.5-0.5B, Explicit CoT=false2026.06 | 0.074 | — | — | — | — | — | |
| NoCoTModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.071 | — | — | — | — | — | |
| iCoTModel=LLaMA-3.2-1B, Explicit CoT=false2026.06 | 0.044 | — | — | — | — | — | |
| PCCOTModel=Qwen2.5-0.5B, Explicit CoT=false2026.06 | 0.041 | — | — | — | — | — |