# Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation

Source: [Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation](<https://www.sota2.com/research/paper/6978471ba2b28e761d87c9f6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-01T05:00:12.177Z

Authors: Mengyuan Liu, Juyi Sheng, Peiming Li, Ziyi Wang, Tianming Xu, Tiantian Xu, Hong Liu

Published: 2026-01-26T17:47:42.000Z

## Abstract

Although Vision--Action (VA) and Vision--Language--Action (VLA) policies have advanced robotic manipulation, their evaluation remains dominated by binary success rates, which obscure process-level differences among executions that complete the same task. We introduce Eval-Actions, a diagnostic evaluation methodology and real-robot benchmark for fine-grained execution-quality assessment of learned manipulation policies. Eval-Actions combines criteria-based Expert Grading (EG), Rank-Guided (RG) labels that align measurable motion indicators with expert rankings, and Chain-of-Thought-style (CoT) annotations that explain observable quality differences. The benchmark contains 13K+ teleoperated and policy-generated real-robot episodes covering 150+ tasks and approximately 52 hours of recordings with RGB-D videos, robot-state trajectories, task descriptions, and success/failure labels. Its densely annotated subset provides EG/RG/CoT supervision for training and evaluation. We further provide AutoEval, a reference multimodal evaluator that predicts quality scores, task outcomes, and diagnostic explanations from RGB temporal evidence and compact kinematic summaries. On the annotated Eval-Actions test split, AutoEval-S achieves Spearman rank correlations (SRCCs) of 0.81 and 0.84 under EG and RG, with success detection accuracies of 90.6% and 91.0%; AutoEval-P reaches 0.70 SRCC under CoT. Analyses of expert consistency, physical-metric baselines, modality ablations, structured generalization, and offline policy ranking show that Eval-Actions provides standardized, interpretable diagnostic signals complementary to success-rate evaluation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Score Prediction on Eval-Actions 1.0 (test)](<https://www.sota2.com/research/sota/score-prediction-on-eval-actions-1-0-test.md>): Score Prediction; Eval-Actions 1.0 (test); SRCC: 0.84.

- [Source Prediction on Eval-Actions 1.0 (test)](<https://www.sota2.com/research/sota/source-prediction-on-eval-actions-1-0-test.md>): Source Prediction; Eval-Actions 1.0 (test); Accuracy: 99.6.

- [Success Prediction on Eval-Actions 1.0 (test)](<https://www.sota2.com/research/sota/success-prediction-on-eval-actions-1-0-test.md>): Success Prediction; Eval-Actions 1.0 (test); Accuracy: 91.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.18723>)

[PDF](<https://arxiv.org/pdf/2601.18723.pdf>)