# Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery

Source: [Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery](<https://www.sota2.com/research/paper/68fa68f41e209131cea1f465>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:49:52.437Z

Authors: Michael Zhang, Xiaotian Cheng, Daniel Copeland, Arjun Desai, Melody Y. Guan, Gabriel A. Brat, Serena Yeung

Published: 2020-12-13T03:10:09.000Z

## Abstract

Open, or non-laparoscopic surgery, represents the vast majority of all operating room procedures, but few tools exist to objectively evaluate these techniques at scale. Current efforts involve human expert-based visual assessment. We leverage advances in computer vision to introduce an automated approach to video analysis of surgical execution. A state-of-the-art convolutional neural network architecture for object detection was used to detect operating hands in open surgery videos. Automated assessment was expanded by combining model predictions with a fast object tracker to enable surgeon-specific hand tracking. To train our model, we used publicly available videos of open surgery from YouTube and annotated these with spatial bounding boxes of operating hands. Our model's spatial detections of operating hands significantly outperforms the detections achieved using pre-existing hand-detection datasets, and allow for insights into intra-operative movement patterns and economy of motion.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2012.06948>)

[PDF](<https://arxiv.org/pdf/2012.06948.pdf>)
