Gestures

Gestures are one of the three primary forms of input on HoloLens. Once you've targeted a hologram with gaze, gestures allow you interact with the hologram. Gesture input lets you interact with your holograms naturally using your hands or, optionally, with a clicker.

In addition to gestures, you can also use voice input to enable interaction with holograms in your app.

Hand recognition

HoloLens recognizes gesture input by tracking the position of either or both hands that are visible to the device. HoloLens sees hands when they are in either the ready state (back of the hand facing you with index finger up) or the pressed state (back of the hand facing you with the index finger down). When hands are in other poses, the HoloLens will ignore them.

HoloLens looks for hand input within a cone in front of the device, known as the gesture frame, which extends above, below, left and right of the display frame where holograms appear. This lets you keep your elbow comfortably at your side while providing hand input. When using the HoloLens Clicker, your hands do not need to be within the gesture frame.

For each hand that HoloLens detects, you can access its position (without orientation) and its pressed state. As the hand nears the edge of the gesture frame, you're also provided with a direction vector, which you can show to the user so they know how to move their hand to get it back where HoloLens can see it.

Interactions

The core interactions on HoloLens are press, release, and bloom.

Press and release

The primary way in which you'll select or activate holograms in your world is the air-tap gesture, which consists of a press followed by a release.

Ready state for hand gestures on HoloLens

To air-tap, make a fist in front of you, with the back of your hand facing you. Your elbow should be bent at your side in a comfortable position. Now, raise your index finger to the sky and then tap, by flexing your index finger down (the press) and then back up (the release).

An air-tap results in a select operation. Other ways to select are using a clicker or the voice command "select." The same air-tap gesture can be used within any holographic app. Think of air-tap (or the "select" keyword) as the equivalent of a mouse click; a universal gesture that you learn once and then apply across all your apps.

Bloom

Bloom is a special system gesture that is used to go back to the Start Menu from within a holographic application. It is similar to pressing the Windows key on a keyboard or the Xbox button on an Xbox controller.

Bloom gesture on HoloLens

To do the bloom gesture, hold out your hand, palm up, with your fingertips together. Then open your hand.

In Windows Holographic, the bloom gesture can be performed with your hand. You can also return to Start by saying "Hey Cortana, Go Home". Note that your app cannot react specifically to bloom gestures, as these are handled by the system.

Gestures

Your app can recognize more than just individual presses and releases. By combining presses and releases with movement of your hand or using a clicker, you can perform more complex gestures as well:

  • Hold: Holding a press beyond the system's Hold threshold.
  • Manipulation: A press, followed by absolute movement of your hand through 3-dimensional world.
  • Navigation: A press, followed by relative movement of your hand or the controller within a 3-dimensional unit cube, potentially on axis-aligned rails. More on this below.

Hold

Hold gestures are similar a touch tap-and-hold and can be used to take a secondary action, such as picking up an object instead of activating it, or showing a context menu.

Manipulation

Manipulation gestures can be used to move, resize or rotate a hologram when you want the hologram to react 1:1 to the user's hand movements. One use for such 1:1 movements is to let the user draw or paint in the world.

As with all gestures, the initial targeting for a Manipulation gesture should be done by Gaze. Once the press starts, any manipulation of the object is then handled by hand movements, freeing the user to look around while they manipulate.

Manipulation may only be performed with hand gestures.

Navigation

Navigation gestures operate like a virtual joystick, and can be used to navigate UI widgets, such as radial menus. You press to start the gesture and then move your hand within a normalized 3D cube, centered around the initial press. You can move your hand along the X, Y or Z axis from a value of -1 to 1, with 0 being the starting point.

Navigation can be used to build velocity-based continuous scrolling or zooming gestures, similar to scrolling a 2D UI by clicking the middle mouse button and then moving the mouse up and down.

Select and hold gesture allows navigation like a virtual joystick

Navigation with rails refers to the ability of recognizing movements in certain axis until certain threshold is reached on that axis. This is only useful, when movement in more than one axis is enabled in an application by the developer, e.g. if an application is configured to recognize navigation gestures across X, Y axis but also specified X axis with rails. In this case system will recognize hand movements across X axis as long as they remain within an imaginary rails (guide) on X axis, if hand movement also occurs Y axis.

Within 2D apps on HoloLens, users can use vertical navigation gestures to scroll, zoom, or drag inside the app. This injects virtual finger touches to the app to simulate touch gestures of the same type. Users can select which of these actions take place by toggling between the tools on the bar above the app, either by selecting the button or saying '<Scroll/Drag/Zoom> Tool'.

See also