Proactive task suggestion on ProactiveAgent (test)
67.55PrecisionTGL
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| TGLBackbone=Claude-Opus-4.7, Evaluation Protocol=+ Ours2026.05 | 67.55 | 97.7 | 72.39 | 32.45 | 79.86 | |
| Claude-Opus-4.7Backbone=Claude-Opus-4.7, Evaluation Protocol=Vanilla2026.05 | 65.71 | 85.86 | 72.53 | 34.29 | 74.44 | |
| TGLBackbone=GPT-5.4-mini, Evaluation Protocol=+ Ours2026.05 | 64.37 | 97.59 | 69.81 | 35.63 | 77.55 | |
| TGLBackbone=Gemma-3-12B-it, Evaluation Protocol=+ Ours2026.05 | 64.02 | 97.58 | 69.53 | 35.98 | 77.3 | |
| TGLBackbone=GPT-5.4, Evaluation Protocol=+ Ours2026.05 | 63.14 | 97.54 | 68.81 | 36.86 | 76.57 | |
| TGLBackbone=DeepSeek-V4-Flash, Evaluation Protocol=+ Ours2026.05 | 63.14 | 97.54 | 68.81 | 36.86 | 76.6 | |
| TGLBackbone=Claude-Sonnet-4.6, Evaluation Protocol=+ Ours2026.05 | 62.61 | 97.52 | 68.38 | 37.39 | 76.2 | |
| TGLBackbone=DeepSeek-V4-Pro, Evaluation Protocol=+ Ours2026.05 | 59.44 | 97.38 | 65.81 | 40.56 | 73.7 | |
| TGLBackbone=Claude-Haiku-4.5, Evaluation Protocol=+ Ours2026.05 | 58.91 | 97.38 | 65.38 | 41.09 | 73.4 | |
| GPT-5.4Backbone=GPT-5.4, Evaluation Protocol=Vanilla2026.05 | 58.88 | 97.72 | 64.38 | 41.12 | 73.47 | |
| TGLBackbone=GPT-4o, Evaluation Protocol=+ Ours2026.05 | 58.73 | 97.36 | 65.24 | 41.27 | 73.23 | |
| TGLBackbone=LLaMA-3.1-8B-Instr., Evaluation Protocol=+ Ours2026.05 | 57.32 | 97.29 | 64.09 | 42.68 | 72.07 | |
| TGLBackbone=Qwen3-8B, Evaluation Protocol=+ Ours2026.05 | 57.32 | 97.31 | 64.09 | 42.68 | 72.14 | |
| TGLBackbone=Qwen2-7B-Instr., Evaluation Protocol=+ Ours2026.05 | 55.56 | 97.22 | 62.66 | 44.44 | 70.68 | |
| TGLBackbone=GPT-4o-mini, Evaluation Protocol=+ Ours2026.05 | 53.97 | 97.14 | 61.37 | 46.03 | 69.38 | |
| GPT-5.4-miniBackbone=GPT-5.4-mini, Evaluation Protocol=Vanilla2026.05 | 52.97 | 94.74 | 63.23 | 47.03 | 67.94 | |
| DeepSeek-V4-FlashBackbone=DeepSeek-V4-Flash, Evaluation Protocol=Vanilla2026.05 | 51.25 | 83.73 | 57.94 | 48.75 | 63.44 | |
| TGLBackbone=Qwen3-4B, Evaluation Protocol=+ Ours2026.05 | 50.97 | 96.97 | 58.94 | 49.03 | 66.77 | |
| Claude-Sonnet-4.6Backbone=Claude-Sonnet-4.6, Evaluation Protocol=Vanilla2026.05 | 49.87 | 71.23 | 58.85 | 50.13 | 58.64 | |
| Qwen2-7B-Proact.Backbone=Qwen2-7B, Evaluation Protocol=Fine-tuned proactive baseline2026.05 | 49.78 | 100 | 50.66 | 50.22 | 66.47 | |
| LLaMA-3.1-8B-Proact.Backbone=LLaMA-3.1-8B, Evaluation Protocol=Fine-tuned proactive baseline2026.05 | 49.76 | 99.06 | 52.86 | 50.24 | 66.25 | |
| GPT-4oBackbone=GPT-4o, Evaluation Protocol=Vanilla2026.05 | 48.15 | 98.11 | 49.78 | 51.85 | 64.6 | |
| Claude-Haiku-4.5Backbone=Claude-Haiku-4.5, Evaluation Protocol=Vanilla2026.05 | 45.03 | 99.31 | 50.93 | 54.97 | 61.89 | |
| Qwen2-7B-Instr.Backbone=Qwen2-7B-Instr., Evaluation Protocol=Vanilla2026.05 | 44 | 98.02 | 43.61 | 56 | 60.74 | |
| DeepSeek-V4-ProBackbone=DeepSeek-V4-Pro, Evaluation Protocol=Vanilla2026.05 | 43.26 | 52.37 | 58.94 | 56.74 | 47.12 | |
| Gemma-3-12B-itBackbone=Gemma-3-12B-it, Evaluation Protocol=Vanilla2026.05 | 42.67 | 91.05 | 46.07 | 57.33 | 58.04 | |
| LLaMA-3.1-8B-Instr.Backbone=LLaMA-3.1-8B-Instr., Evaluation Protocol=Vanilla2026.05 | 38.16 | 98.86 | 39.06 | 61.84 | 55.06 | |
| GPT-4o-miniBackbone=GPT-4o-mini, Evaluation Protocol=Vanilla2026.05 | 35.28 | 100 | 36.12 | 64.73 | 52.15 | |
| Qwen3-4BBackbone=Qwen3-4B, Evaluation Protocol=Vanilla2026.05 | 23.89 | 100 | 25.75 | 76.11 | 38.54 | |
| Qwen3-8BBackbone=Qwen3-8B, Evaluation Protocol=Vanilla2026.05 | 15.07 | 99.07 | 19.17 | 84.93 | 26.14 |