Reasoning on BBH (Accuracy)
74.99BBH AccuracyNTele-R1-32B
Evaluation Results
| Method | Links | |
|---|---|---|
| NTele-R1-32B2025.08 | 74.99 | |
| DS-32B2025.08 | 72.74 | |
| OuroModel Version=1.4B R4, Architecture=LoopLM, # Params=1.4B, # Tokens=7.7T, recurrent steps=42025.10 | 71.02 | |
| Qwen3Model Version=4B, Architecture=Dense, # Params=4.0B, # Tokens=36T2025.10 | 70.95 | |
| Gemma3Model Version=4B, Architecture=Dense, # Params=4.0B, # Tokens=4T2025.10 | 66.32 | |
| CODABackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 65 | |
| GradNormBackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 63.8 | |
| PCGradBackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 63.6 | |
| GREATSBackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 63.4 | |
| DoReMiBackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 63.1 | |
| Full dataBackbone=Qwen-2.5-7B, Data Budget=100%2026.07 | 62.8 | |
| TemperatureBackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 62.7 | |
| ProportionalBackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 62.3 | |
| Uniform mixingBackbone=Qwen-2.5-7B, Data Budget=50%2026.07 | 62 | |
| TTT+CT-KVTraining time per task (s)=172025.07 | 58.2 | |
| CT-KVTraining time per task (s)=72025.07 | 57.9 | |
| TTTTraining time per task (s)=142025.07 | 57.8 | |
| CT-PromptTraining time per task (s)=142025.07 | 56.3 | |
| CODABackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 56.3 | |
| Qwen2.5Model Version=3B, Architecture=Dense, # Params=3.0B, # Tokens=18T2025.10 | 55.37 | |
| GradNormBackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 55.1 | |
| PCGradBackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 54.9 | |
| GREATSBackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 54.7 | |
| DoReMiBackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 54.4 | |
| Full dataBackbone=LLaMA-3.1-8B, Data Budget=100%2026.07 | 54.2 | |
| TemperatureBackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 54 | |
| Qwen3Model Version=1.7B, Architecture=Dense, # Params=1.7B, # Tokens=36T2025.10 | 53.51 | |
| ProportionalBackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 53.5 | |
| Uniform mixingBackbone=LLaMA-3.1-8B, Data Budget=50%2026.07 | 53.2 | |
| Prefix TuningTraining time per task (s)=9, m=# demo2025.07 | 52.8 | |
| Prefix TuningTraining time per task (s)=7, m=322025.07 | 52.7 | |
| DoRATraining time per task (s)=92025.07 | 52.6 | |
| LoRATraining time per task (s)=92025.07 | 51.7 | |
| Prompt TuningTraining time per task (s)=7, m=322025.07 | 50.8 | |
| ICLTraining time per task (s)=0, Evaluation protocol=In-Context Learning2025.07 | 50.4 | |
| Prompt TuningTraining time per task (s)=16, m=# demo2025.07 | 47.5 | |
| Rank-Stabilized LoRATraining time per task (s)=82025.07 | 46.7 | |
| Qwen2.5Model Version=1.5B, Architecture=Dense, # Params=1.5B, # Tokens=18T2025.10 | 43.66 | |
| Zero-ShotTraining time per task (s)=0, Evaluation protocol=Zero-Shot2025.07 | 40.9 | |
| MERGEvolveSetting=single-task, Number of runs=52026.06 | 39.53 | |
| Llama3.2Model Version=3B, Architecture=Dense, # Params=3.0B, # Tokens=9T2025.10 | 39.45 | |
| EMMSetting=single-task, Number of runs=52026.06 | 39.35 | |
| LoraHubSetting=single-task, Number of runs=52026.06 | 39.3 | |
| Pack of LLMsSetting=single-task, Number of runs=52026.06 | 39.2 | |
| Model SwarmsSetting=single-task, Number of runs=52026.06 | 38.58 | |
| TIESSetting=single-task, Number of runs=52026.06 | 36.2 | |
| Expert FusionSetting=single-task, Number of runs=52026.06 | 31.82 | |
| Llama3.2Model Version=1.2B, Architecture=Dense, # Params=1.0B, # Tokens=9T2025.10 | 30.72 | |
| Gemma3Model Version=1B, Architecture=Dense, # Params=1.0B, # Tokens=2T2025.10 | 30.26 | |
| Best Single ExpertSetting=single-task, Number of runs=52026.06 | 30.22 | |
| Data MergeSetting=single-task, Number of runs=52026.06 | 25.98 |