Gesture design

Users will use gaze, gesture and voice (GGV) as the primary means of interacting with your content on HoloLens. Since Gaze is the targeting mechanism, users will use this to target content via Cursors. Gesture and voice are the intention mechanisms. Users will use these to indicate that they are ready to interact with the content they are targeting. The interaction model on HoloLens is GG and GV. Users will use gaze with either gestures or voice to complete their interactions. While designing apps, you should consider how you can make these interactions work together well.

Gesture

Users use gestures to take action. There are two types of Gestures supported by HoloLens. Discrete gestures like air-tap and Continuous gestures like navigation and manipulation.

Discrete Gesture Discrete gestures are gestures which happen quickly. Air-tap is the best example of this gesture. In the air-tap gesture, the user presses their finger down and back again. It is similar to the click action in mouse when the user presses the button down and up.

Continuous Gestures Continuous gestures are gestures which happen over time. They typically start when the user does the pressed gesture and end with the release gesture. A good example of how continuous gestures are used is the scroll mechanism that users use when scrolling content in apps like Edge. The users go into scroll mode by holding their hand in the pressed gesture. As they move their hand in this gesture, the content scrolls. When they reach the page they wanted, the user does the release gesture to get out of the scroll mode.

Remember that when the user performs a gesture, they intend to take action on the object that they are targeting. This becomes interesting for continuous gestures where the gestures may take significant time to complete. In such cases, the targeting should be locked to the content the user was gazing at when they started the gesture. In the HoloLens shell, the cursor will lock to the application while the user is scrolling.

Gesture detected feedback. HoloLens detects the users hand through a camera on the front of the device. For the hand to be detected, the following conditions must be met.

  1. The hand is in the gesture frame
  2. The hand is in the ready state

It is important for the user to know that their hands are being detected by HoloLens as that makes them more confident that their gestures will be detected. In the HoloLens shell, the cursor changes from a solid circle to a larger but hollow circle to indicate that the users hand is being detected. Provide real time feedback in your app to indicate when the hand are being detected and when not. The user will then correct themselves automatically and be more confident in their gestures.

Handling gesture frame While the gesture detected feedback works well for discrete gestures, it is not enough for continuous gestures. If a users hand leaves the gesture frame in the middle of a continuous gesture, the gesture will be cancelled. It is important therefore for users to know that their hand in about to leave the frame before it happens. HoloLens provides events for letting the application know when the hand is getting close to the frame's edge and also the direction in which the hand should be moved to be back in the center. Apps can use this information to provide the right feedback to users.

See also