Categories Companies Research

Loading the SOTA2 catalog…

Explore

Categories Companies Research

Contribute

Join SOTA2 Contact

SOTA2

© 2026 SOTA2

Build in Bellevue, USA

WizWand is now SOTA2 Research

Papers Benchmarks Datasets Tasks

ShellOps dataset and benchmarks · SOTA2 Research

Research Datasets

ShellOps

Benchmarks

Task NameDataset NameSOTA MethodSort

Task NameDataset NameSOTA ResultTrendResults

Macro-average Exact Match

ShellOps-Pro

53.3Macro-average Exact Match Accuracy

36

ShellOps

52Combined Score

9

ShellOps

58.3Diff Recall

9

ShellOps

49.1LLM Judge Accuracy

9

Agentic Task Solving

ShellOps

0.462Pass@3

9

Hybrid Operations

ShellOps

24.6Exact Match

9

ShellOps

26.5Exact Match

9

String Extraction

ShellOps

48.5Exact Match

9

Popular tasks

Agentic task solving Macro-average Exact Match String Extraction File Editing Hybrid Operations

Follow for updates

Join SOTA2 Contact