# Smart Home Agent Interaction on SmartHome-Bench

Source: [Smart Home Agent Interaction on SmartHome-Bench](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T10:41:13.340Z

Evaluation of agent performance on smart home interaction tasks using the SmartHome-Bench dataset, measured by quality scores (QT1-QT5), overall score, and resource usage (tool calls, tokens).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Smart Home Agent Interaction |
| Dataset (including any stated split) | SmartHome-Bench |
| Metric | Quality Task 1 (QT1) |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 14 of 14 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Quality Task 1 (QT1)](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Quality%20Task%201%20(QT1)>): Higher is better; unit: PERCENT.

- [Quality Task 2 (QT2)](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Quality%20Task%202%20(QT2)>): Higher is better; unit: PERCENT.

- [Quality Task 3 (QT3)](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Quality%20Task%203%20(QT3)>): Higher is better; unit: PERCENT.

- [Quality Task 4 (QT4)](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Quality%20Task%204%20(QT4)>): Higher is better; unit: PERCENT.

- [Quality Task 5 (QT5)](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Quality%20Task%205%20(QT5)>): Higher is better; unit: PERCENT.

- [Overall Score](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Overall%20Score>): Higher is better; unit: PERCENT.

- [Avg. Tool Calls](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Avg.%20Tool%20Calls>): Lower is better; unit: SCALAR.

- [Avg. Tokens (K)](<https://www.sota2.com/research/sota/smart-home-agent-interaction-on-smarthome-bench.md?metric=Avg.%20Tokens%20(K)>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Quality Task 1 (QT1) | Result date | Report paper |
| --- | --- | --- | --- |
| HomeFlow-RL | 96.22 | Not recorded | 6a1e6407dbd7150506f85619 |
| HomeFlow-SFT | 94.99 | Not recorded | 6a1e6407dbd7150506f85619 |
| HomeFlow-SFT | 94.99 | Not recorded | 6a1e6407dbd7150506f85619 |
| HomeFlow-RL | 94.72 | Not recorded | 6a1e6407dbd7150506f85619 |
| DeepSeek-V4-Flash | 93.14 | Not recorded | 6a1e6407dbd7150506f85619 |
| Role-play | 91.56 | Not recorded | 6a1e6407dbd7150506f85619 |
| GLM-5 | 91.03 | Not recorded | 6a1e6407dbd7150506f85619 |
| Role-play | 89.71 | Not recorded | 6a1e6407dbd7150506f85619 |
| Qwen-3.5 Plus | 89.45 | Not recorded | 6a1e6407dbd7150506f85619 |
| Claude-4.6-Sonnet | 88.65 | Not recorded | 6a1e6407dbd7150506f85619 |
| MiniMax M2.7 | 73.88 | Not recorded | 6a1e6407dbd7150506f85619 |
| GPT-5.5 | 72.82 | Not recorded | 6a1e6407dbd7150506f85619 |
| Qwen3 | 38 | Not recorded | 6a1e6407dbd7150506f85619 |
| Qwen3 | 27.7 | Not recorded | 6a1e6407dbd7150506f85619 |

### HomeFlow-RL

Quality Task 1 (QT1): 96.22. Result ID: 6a1eb32ffbc373592c199b93.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: {"Model Size":"4B"}. This record does not assert independent reproduction.

### HomeFlow-SFT

Quality Task 1 (QT1): 94.99. Result ID: 6a1eb32ffbc373592c199b92.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: {"Model Size":"4B"}. This record does not assert independent reproduction.

### HomeFlow-SFT

Quality Task 1 (QT1): 94.99. Result ID: 6a1eb32ffbc373592c199b96.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: {"Model Size":"8B"}. This record does not assert independent reproduction.

### HomeFlow-RL

Quality Task 1 (QT1): 94.72. Result ID: 6a1eb32ffbc373592c199b97.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: {"Model Size":"8B"}. This record does not assert independent reproduction.

### DeepSeek-V4-Flash

Quality Task 1 (QT1): 93.14. Result ID: 6a1eb32ffbc373592c199b8e.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### Role-play

Quality Task 1 (QT1): 91.56. Result ID: 6a1eb32ffbc373592c199b95.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76921e209131cea6dcfe.md>)

Recorded evaluation configuration: {"Model Size":"8B"}. This record does not assert independent reproduction.

### GLM-5

Quality Task 1 (QT1): 91.03. Result ID: 6a1eb32ffbc373592c199b8d.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/699547d509c33900fcd6d78c.md>)

[Implementation](<https://github.com/zai-org/GLM-5>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### Role-play

Quality Task 1 (QT1): 89.71. Result ID: 6a1eb32ffbc373592c199b91.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76921e209131cea6dcfe.md>)

Recorded evaluation configuration: {"Model Size":"4B"}. This record does not assert independent reproduction.

### Qwen-3.5 Plus

Quality Task 1 (QT1): 89.45. Result ID: 6a1eb32ffbc373592c199b8c.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### Claude-4.6-Sonnet

Quality Task 1 (QT1): 88.65. Result ID: 6a1eb32ffbc373592c199b8b.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### MiniMax M2.7

Quality Task 1 (QT1): 73.88. Result ID: 6a1eb32ffbc373592c199b8f.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### GPT-5.5

Quality Task 1 (QT1): 72.82. Result ID: 6a1eb32ffbc373592c199b8a.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction.

### Qwen3

Quality Task 1 (QT1): 38. Result ID: 6a1eb32ffbc373592c199b90.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Model Size":"4B"}. This record does not assert independent reproduction.

### Qwen3

Quality Task 1 (QT1): 27.7. Result ID: 6a1eb32ffbc373592c199b94.

[Reporting paper](<https://www.sota2.com/research/paper/6a1e6407dbd7150506f85619.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Model Size":"8B"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
