Gestures in Unity

Gestures are the second "G" in Gaze, Gesture, and Voice. Unity provides both low level access (raw position and velocity information) and a high level gesture recognizer that exposes more complex gesture events (for example: tap, double tap, hold, manipulation and navigation).

Gesture Input

Namespace: UnityEngine.VR.WSA.Input

Types: GestureRecognizer, GestureSettings, InteractionSourceKind

These high level gestures are generated by input sources. Each Gesture event provides the SourceKind for the input as well as the targeting head ray at the time of the event. Some events provide additional context specific information.

There are only a few steps required to capture gestures using a Gesture Recognizer:

  1. Create a new Gesture Recognizer
  2. Specify which gestures to watch for
  3. Subscribe to events for those gestures
  4. Start capturing gestures

Create a new Gesture Recognizer

To use the GestureRecognizer, you must have created a GestureRecognizer:

GestureRecognizer recognizer = new GestureRecognizer();

Specify which gestures to watch for

Specify which gestures you are interested in via SetRecognizableGestures():

recognizer.SetRecognizableGestures(GestureSettings.Tap | GestureSettings.Hold);

Subscribe to events for those gestures

Subscribe to events for the gestures you are interested in.

recognizer.TappedEvent += MyTapEventHandler;
recognizer.HoldEvent += MyHoldEventHandler;

Note: Navigation and Manipulation gestures are mutually exclusive on an instance of a GestureRecognizer.

Start capturing gestures

By default, a GestureRecognizer does not monitor input until StartCapturingGestures() is called. It is possible that a gesture event may be generated after StopCapturingGestures() is called if input was performed before the frame where StopCapturingGestures() was processed. Because of this, it is reliable if you want to start and stop gesture monitoring depending on which object the player is currently gazing at.

recognizer.StartCapturingGestures();

Stop capturing gestures

To stop gesture recognition:

recognizer.StopCapturingGestures();

Removing a gesture recognizer

Remember to unsubscribe from subscribed events before destroying a GestureRecognizer object.

void OnDestroy()
{
    recognizer.TappedEvent -= MyTapEventHandler;
    recognizer.HoldEvent -= MyHoldEventHandler;
}

Interaction Input

Namespace: UnityEngine.VR.WSA.Input

Types: InteractionManager, InteractionSourceState, InteractionSource, InteractionSourceProperties, InteractionSourceKind, InteractionSourceLocation

Low level spatial input allows you to get much more detailed information on the input mechanism such as its position and velocity in the world.

How to start handling an interaction event

Using interaction input is easy:

  1. Register for a InteractionManager input event
  2. Handle the event

It's also easy to stop handling:

  1. Unregister the interaction event

Register for the interaction event

For each type of interaction event that you are interested in, you need to subscribe to it.

InteractionManager.SourcePressed += InteractionManager_SourcePressed;

Handle the interaction event

Once you have subscribed to an interaction event, you will get the callback when appropriate. In the SourcePressed example, this will be after the source was detected and before it is released or lost.

void InteractionManager_SourcePressed(InteractionSourceState state)
{
    // state has information about:
       // targeting head ray at the time when the event was triggered
       // whether the source is pressed or not
       // properties like position, velocity, source loss risk
       // source id (which hand id for example) and source kind like hand, voice, controller or other
}

How to stop handling an event

You need to stop handling an event when you are no longer interested in the event or you are destroying the object that has subscribed to the event.

Unregister for the interaction event

To stop handling the event, you unsubscribe from the event.

InteractionManager.SourcePressed -= InteractionManager_SourcePressed;

Input Source Change Events

These events describe when an input source is:

  • detected (becomes active)
  • lost (becomes inactive)
  • updates (moves or otherwise changes some state)
  • is pressed (tap, button press, or select uttered)
  • is released (end of a tap, button released, or end of select uttered)

Input Source State

Each of these events exposes the input InteractionSourceState. The input InteractionSourceState represents an input source at a moment in time. The InteractionSourceState exposes info such as:

  • whether it is in a pressed state
  • InteractionSourceProperties which contains the InteractionSourceLocation which can be used to find the position and velocity of the input source at that point in time
  • the headRay which can be used to determine what the user was looking at when the event occurred since it can be from a previous frame
  • the InteractionSource of the input which can be of varying InteractionSourceKind's such as Hand, Voice, Controller, or Other.

Example

using UnityEngine.VR.WSA.Input;

void Start ()
{
    InteractionManager.SourceDetected += InteractionManager_SourceDetected;
    InteractionManager.SourceUpdated += InteractionManager_SourceUpdated;
    InteractionManager.SourceLost += InteractionManager_SourceLost;
    InteractionManager.SourcePressed += InteractionManager_SourcePressed;
    InteractionManager.SourceReleased += InteractionManager_SourceReleased;
}

void OnDestroy()
{
    InteractionManager.SourceDetected -= InteractionManager_SourceDetected;
    InteractionManager.SourceUpdated -= InteractionManager_SourceUpdated;
    InteractionManager.SourceLost -= InteractionManager_SourceLost;
    InteractionManager.SourcePressed -= InteractionManager_SourcePressed;
    InteractionManager.SourceReleased -= InteractionManager_SourceReleased;
}

void InteractionManager_SourceDetected(InteractionSourceState state)
{
    // Source was detected
    // state has the current state of the source including id, position, kind, etc.
}

void InteractionManager_SourceLost(InteractionSourceState state)
{
    // Source was lost. This will be after a SourceDetected event and no other events for this source id will occur until it is Detected again
    // state has the current state of the source including id, position, kind, etc.
}

void InteractionManager_SourceUpdated(InteractionSourceState state)
{
    // Source was updated. The source would have been detected before this point
    // state has the current state of the source including id, position, kind, etc.
}

void InteractionManager_SourcePressed(InteractionSourceState state)
{
    // Source was pressed. This will be after the source was detected and before it is released or lost
    // state has the current state of the source including id, position, kind, etc.
}

void InteractionManager_SourceReleased(InteractionSourceState state)
{
    // Source was released. The source would have been detected and pressed before this point. This event will not fire if the source is lost
    // state has the current state of the source including id, position, kind, etc.
}

See Also