# Semantic Interaction in Augmented Reality Environments for Microsoft HoloLens

Source: [Semantic Interaction in Augmented Reality Environments for Microsoft HoloLens](<https://www.sota2.com/research/paper/68fa6b6b1e209131cea2e396>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:48:06.915Z

Authors: Peer Sch\\"uett, Max Schwarz, Sven Behnke

Published: 2021-11-18T14:58:04.000Z

## Abstract

Augmented Reality is a promising technique for human-machine interaction. Especially in robotics, which always considers systems in their environment, it is highly beneficial to display visualizations and receive user input directly in exactly that environment. We explore this idea using the Microsoft HoloLens, with which we capture indoor environments and display interaction cues with known object classes. The 3D mesh recorded by the HoloLens is annotated on-line, as the user moves, with semantic classes using a projective approach, which allows us to use a state-of-the-art 2D semantic segmentation method. The results are fused onto the mesh; prominent object segments are identified and displayed in 3D to the user. Finally, the user can trigger actions by gesturing at the object. We both present qualitative results and analyze the accuracy and performance of our method in detail on an indoor dataset.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2112.05846>)

[PDF](<https://arxiv.org/pdf/2112.05846.pdf>)