Free Hand-Based 3D Interaction in Optical See-Through Augmented Reality Using Leap Motion

Abstract

In augmented reality environments, the natural hand interaction between a virtual object and the user is a major issue to manipulate a rendered object in a convenient way. Microsoft’s HoloLens (Microsoft 2018) is an innovative augmented reality (AR) device that has provided an impressive experience for the user. However, the gesture interactions offered to the user are very limited. HoloLens currently recognizes two core component gestures: Air tap and Bloom. To solve this issue, we propose to integrate a Leap Motion Controller (LMC) within the HoloLens device (Figure 1). We thus used 3D hand and finger tracking provided by the LMC to propose new free hand-based interaction more natural and intuitive. We implemented three fully 3D techniques for selection, translation and rotation manipulation. In this work, we first investigated how to combine the two devices to get them working together in real time, and then we evaluated the proposed 3D hand interactions.