Mathematical Reasoning on GSM8K (Greedy, Pass@8)
66.9Greedy AccuracySFT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SFTModel=Llama-3.2-3B, Variant=w/ CoT, TTFT (ms)=754.42026.05 | 66.9 | 78.3 | |
| RiMModel=Llama-3.2-3B, Variant=Any block, TTFT (ms)=27.92026.05 | 57.3 | 71.8 | |
| RiMModel=Llama-3.2-1B, Variant=Any block, TTFT (ms)=16.12026.05 | 51.4 | 76.8 | |
| CoconutBackbone=Llama-3.2-1B2026.05 | 50.6 | — | |
| RiMBackbone=Llama-3.2-3B2026.05 | 49.4 | — | |
| SFTModel=Llama-3.2-1B, Variant=w/ CoT, TTFT (ms)=420.32026.05 | 49.1 | 64.7 | |
| RiMModel=Llama-3.2-3B, Variant=Final block, TTFT (ms)=27.92026.05 | 48.8 | 58.8 | |
| DARTBackbone=Llama-3.2-3B2026.05 | 46.6 | — | |
| CoconutBackbone=Llama-3.2-3B2026.05 | 43.6 | — | |
| RiMBackbone=Llama-3.2-1B2026.05 | 43.1 | — | |
| DARTBackbone=Llama-3.2-1B2026.05 | 42.6 | — | |
| CoconutModel=Llama-3.2-3B, Variant=w/o Stage 0, TTFT (ms)=188.82026.05 | 42.2 | 56.8 | |
| RiMModel=Llama-3.2-1B, Variant=Final block, TTFT (ms)=16.12026.05 | 42.1 | 56.1 | |
| CoconutModel=Llama-3.2-3B, Variant=w/ Stage 0, TTFT (ms)=188.82026.05 | 41.3 | 55.5 | |
| SFTModel=GPT-2, Variant=w/ CoT, TTFT (ms)=213.72026.05 | 39.8 | 57 | |
| RiMModel=GPT-2, Variant=Any block, TTFT (ms)=7.62026.05 | 39.5 | 78.1 | |
| CoconutModel=Llama-3.2-1B, Variant=w/ Stage 0, TTFT (ms)=108.32026.05 | 36.9 | 51.1 | |
| SFTModel=Llama-3.2-3B, Variant=w/o CoT, TTFT (ms)=27.92026.05 | 36.2 | 45.9 | |
| RiMBackbone=GPT-22026.05 | 35.5 | — | |
| CoconutBackbone=GPT-22026.05 | 34.1 | — | |
| RiMModel=GPT-2, Variant=Final block, TTFT (ms)=7.62026.05 | 33.6 | 49.1 | |
| CoconutModel=GPT-2, Variant=w/ Stage 0, TTFT (ms)=53.42026.05 | 31.1 | 45 | |
| CoconutModel=Llama-3.2-1B, Variant=w/o Stage 0, TTFT (ms)=108.32026.05 | 30.1 | 46.1 | |
| DARTBackbone=GPT-22026.05 | 24.7 | — | |
| SFTModel=Llama-3.2-1B, Variant=w/o CoT, TTFT (ms)=16.12026.05 | 23.9 | 41.7 | |
| CoconutModel=GPT-2, Variant=w/o Stage 0, TTFT (ms)=53.42026.05 | 22.3 | 40.6 | |
| SFTModel=GPT-2, Variant=w/o CoT, TTFT (ms)=7.62026.05 | 15.4 | 33.3 |