Item crafting on TextCraft (test)
71Success RateFed-SE
Evaluation results
| Method | Links | |
|---|---|---|
| Fed-SEBase Model=Qwen3-8B2025.12 | 71 | |
| LocalBase Model=Qwen3-8B2025.12 | 70 | |
| Fed-SEBase Model=Qwen2.5-7B2025.12 | 67 | |
| LocalBase Model=Qwen2.5-7B2025.12 | 63 | |
| CentralizedBase Model=Qwen3-8B2025.12 | 63 | |
| FedITBase Model=Qwen3-8B2025.12 | 61 | |
| CentralizedBase Model=Qwen2.5-7B2025.12 | 59 | |
| ADaPT (Backend: GPT-4o)Backend=GPT-4o, # Parameters=1.8T*, Open Weight?=No2025.12 | 58 | |
| ADaPT (Backend: Mistral Small 3)Backend=Mistral Small 3, # Parameters=24B, Open Weight?=Yes2025.12 | 58 | |
| Fed-SEBase Model=Qwen2.5-3B2025.12 | 58 | |
| SCOPE# Parameters=11.04M, Open Weight?=-2025.12 | 56 | |
| FedITBase Model=Qwen2.5-7B2025.12 | 54 | |
| FedITBase Model=Qwen2.5-3B2025.12 | 54 | |
| CentralizedBase Model=Qwen3-1.7B2025.12 | 54 | |
| ADaPT (Backend: GPT-3.5)Backend=GPT-3.5, # Parameters=175B, Open Weight?=No2025.12 | 52 | |
| Fed-SEBase Model=Llama2-7B2025.12 | 52 | |
| FedITBase Model=Llama2-7B2025.12 | 49 | |
| CentralizedBase Model=Qwen2.5-3B2025.12 | 48 | |
| LocalBase Model=Qwen2.5-3B2025.12 | 46 | |
| FedITBase Model=Qwen3-1.7B2025.12 | 46 | |
| Fed-SEBase Model=Qwen3-1.7B2025.12 | 46 | |
| ADaPT (Backend: GPT-4o mini)Backend=GPT-4o mini, # Parameters=8B*, Open Weight?=No2025.12 | 43 | |
| LocalBase Model=Qwen3-1.7B2025.12 | 42 | |
| CentralizedBase Model=Llama2-7B2025.12 | 36 | |
| LocalBase Model=Llama2-7B2025.12 | 33 | |
| Pre-TrainedBase Model=Qwen3-1.7B2025.12 | 15 | |
| ADaPT (Backend: DeepSeek-R1-Distill-Qwen-32B)Backend=DeepSeek-R1-Distill-Qwen-32B, # Parameters=32B, Open Weight?=Yes2025.12 | 13 | |
| Pre-TrainedBase Model=Qwen2.5-7B2025.12 | 4 | |
| Pre-TrainedBase Model=Qwen2.5-3B2025.12 | 1 | |
| Pre-TrainedBase Model=Qwen3-8B2025.12 | 1 | |
| ADaPT (Backend: Claude-3 Haiku)Backend=Claude-3 Haiku, # Parameters=20B*, Open Weight?=No2025.12 | 0 | |
| Pre-TrainedBase Model=Llama2-7B2025.12 | 0 |