Motion controllers are hardware accessories that allow users to take action in mixed reality. An advantage of motion controllers over gestures is that the controllers have a precise position in space, allowing for fine grained interaction with digital objects. For Windows Mixed Reality immersive headsets, motion controllers are the primary way that users will take action in their world.
Windows Mixed Reality motion controllers offer precise and responsive tracking of movement in your field of view using the sensors in the immersive headset, meaning there is no need to install hardware on the walls in your space. These motion controllers will offer the same ease of setup and portability as Windows Mixed Reality immersive headsets. Our device partners plan to market and sell these controllers on retail shelves this holiday.
Before you begin
You will need:
A set of two motion controllers.
Four AA batteries.
A PC capable of Bluetooth 4.0.
Update Windows and Unity for motion controller support:
If your headset shows a different driver version, then click Update Driver and Search automatically for updated driver software
Insert 2 AA batteries into the back of the controller. Leave the battery cover off for now.
If you're using a USB Bluetooth Adapter instead of a built-in Bluetooth radio, please review the Bluetooth Best Practices before proceeding
Press and hold the controller's Windows button to turn on the controller, release once it buzzes.
Press and hold the pairing button (tab in the battery compartment) until the LEDs begin pulsing. Reattach battery cover.
Open Windows Settings -> Bluetooth & other devices -> Add Bluetooth or other device -> Bluetooth and wait for "Motion controller - Left" or "Motion controller - Right" to appear. Select to pair. Controller will vibrate once when connected.
Repeat steps 1-4 for the second controller.
Updating controller firmware
If an immersive headset is connected to your PC, and new controller firmware is available, the firmware will be pushed to your motion controllers automatically the next time they're turned on. Controller firmware updates are indicated by a pattern of illuminating LED quadrants in a circular motion, and take 1-2 minutes.
After the firmware update completes, the controllers will reboot and reconnect.
Both controllers should be connected now.
Verify your controllers work properly:
Launch Mixed Reality Portal and enter the Cliff House.
Windows Mixed Reality supports two key models for interaction, gaze and commit and point and commit:
With gaze and commit, users target an object with their gaze and then select objects with hand air-taps, a gamepad, a clicker or their voice.
With point and commit, a user can aim a pointing-capable motion controller at the target object and then select objects with the controller's trigger.
Apps that support pointing with motion controllers should also enable gaze-driven interactions where possible, to give users choice in what input devices they use.
Managing recoil when pointing
When using motion controllers to point and commit, your users will use the controller to target and then take action by pulling its trigger. Users who pull the trigger vigorously may end up aiming the controller higher at the end of their trigger pull than they'd intended.
To manage any such recoil that may occur when users pull the trigger, your app can snap its targeting ray when the trigger's analog axis value rises above 0.0. You can then then take action using that targeting ray a few frames later once the trigger value reaches 1.0, as long as the final press occurs within a short time window. When using the higher-level composite Tap gesture, Windows will manage this targeting ray capture and timeout for you.
Grip pose vs. pointing pose
Windows Mixed Reality supports motion controllers in a variety of form factors, with each controller's design differing in its relationship between the user's hand position and the natural "forward" direction that apps should use for pointing when rendering the controller.
To better represent these controllers, there are two kinds of poses you can investigate for each interaction source, the grip pose and the pointer pose.
The grip pose represents the location of either the palm of a hand detected by a HoloLens, or the palm holding a motion controller.
On immersive headsets, the grip pose is best used to render the user's hand or an object held in the user's hand, such as a sword or gun. The grip pose is also used when visualizing a motion controller, as the renderable model provided by Windows for a motion controller uses the grip pose as its origin and center of rotation.
The grip pose is defined specifically as follows:
The grip position: The palm centroid when holding the controller naturally, adjusted left or right to center the position within the grip. On the Windows Mixed Reality motion controller, this position generally aligns with the Grasp button.
The grip orientation's Right axis: When you completely open your hand to form a flat 5-finger pose, the ray that is normal to your palm (forward from left palm, backward from right palm)
The grip orientation's Forward axis: When you close your hand partially (as if holding the controller), the ray that points "forward" through the tube formed by your non-thumb fingers.
The grip orientation's Up axis: The Up axis implied by the Right and Forward definitions.
The pointer pose represents the tip of the controller pointing forward.
The system-provided pointer pose is best used to raycast when you are rendering the controller model itself. If you are rendering some other virtual object in place of the controller, such as a virtual gun, you should point with a ray that is most natural for that virtual object, such as a ray that travels along the barrel of the app-defined gun model. Because users can see the virtual object and not the physical controller, pointing with the virtual object will likely be more natural for those using your app.
Controller tracking state
Like the headsets, the Windows Mixed Reality motion controller requires no setup of external tracking sensors. Instead, the controllers are tracked by sensors in the headset itself.
If the user moves the controllers out of the headset's field of view, in most cases Windows will continue to infer controller positions and provide them to the app. When the controller has lost visual tracking for long enough, the controller's positions will drop to approximate-accuracy positions.
At this point, the system will body-lock the controller to the user, tracking the user's position as they move around, while still exposing the controller's true orientation using its internal orientation sensors. Many apps that use controllers to point at and activate UI elements can operate normally while in approximate accuracy without the user noticing.
The best way to get a feel for this is to try it yourself. Check out this video with examples of immersive content that works with motion controllers across various tracking states:
Reasoning about tracking state explicitly
Apps that wish to treat positions differently based on tracking state may go further and inspect properties on the controller's state, such as SourceLossRisk and PositionAccuracy:
High accuracy (at risk of losing)
These motion controller tracking states are defined as follows:
High accuracy: While the motion controller is within the headset's field of view, it will generally provide high-accuracy positions, based on visual tracking. Note that a moving controller that momentarily leaves the field of view or is momentarily obscured from the headset sensors (e.g. by the user's other hand) will continue to return high-accuracy poses for a short time, based on inertial tracking of the controller itself.
High accuracy (at risk of losing): When the user moves the motion controller past the edge of the headset's field of view, the headset will soon be unable to visually track the controller's position. The app knows when the controller has reached this FOV boundary by seeing the SourceLossRisk reach 1.0. At that point, the app may choose to pause controller gestures that require a steady stream of very high-quality poses.
Approximate accuracy: When the controller has lost visual tracking for long enough, the controller's positions will drop to approximate-accuracy positions. At this point, the system will body-lock the controller to the user, tracking the user's position as they move around, while still exposing the controller's true orientation using its internal orientation sensors. Many apps that use controllers to point at and activate UI elements can operate as normal while in approximate accuracy without the user noticing. Apps with heavier input requirements may choose to sense this drop from High accuracy to Approximate accuracy by inspecting the PositionAccuracy property, for example to give the user a more generous hitbox on off-screen targets during this time.
No position: While the controller can operate at approximate accuracy for a long time, sometimes the system knows that even a body-locked position is not meaningful at the moment. For example, a controller that was just turned on may have never been observed visually, or a user may put down a controller that's then picked up by someone else. At those times, the system will not provide any position to the app, and TryGetPosition will return false.
Interactions: Low-level spatial input
The core interactions across hands and motion controllers are Select, Menu, Grasp, Touchpad, Thumbstick, and Home.
Select is the primary interaction to activate a hologram, consisting of a press followed by a release. For motion controllers, you perform a Select press using the controller's trigger. Other ways to perform a Select are by speaking the voice command "Select". The same select interaction can be used within any app. Think of Select as the equivalent of a mouse click, a universal action that you learn once and then apply across all your apps.
Menu is the secondary interaction for acting on an object, used to pull up a context menu or take some other secondary action. With motion controllers, you can take a menu action using the controller's menu button. (i.e. the button with the hamburger "menu" icon on it)
Grasp is how users can directly take action on objects at their hand to manipulate them. With motion controllers, you can do a grasp action by squeezing your fist tightly. A motion controller may detect a Grasp with a grab button, palm trigger or other sensor.
Touchpad allows the user to adjust an action in two dimensions along the surface of a motion controller's touchpad, committing the action by clicking down on the touchpad. Touchpads provide a pressed state, touched state and normalized XY coordinates. X and Y range from -1 to 1 across the range of the circular touchpad, with a center at (0, 0). For X, -1 is on the left and 1 is on the right. For Y, -1 is on the bottom and 1 is on the top.
Thumbstick allows the user to adjust an action in two dimensions by moving a motion controller's thumbstick within its circular range, committing the action by clicking down on the thumbstick. Thumbsticks also provide a pressed state and normalized XY coordinates. X and Y range from -1 to 1 across the range of the circular touchpad, with a center at (0, 0). For X, -1 is on the left and 1 is on the right. For Y, -1 is on the bottom and 1 is on the top.
Home is a special system action that is used to go back to the Start Menu. It is similar to pressing the Windows key on a keyboard or the Xbox button on an Xbox controller. You can go home by pressing the Windows button on a motion controller. Note, you can also always return to Start by saying "Hey Cortana, Go Home". Apps cannot react specifically to home actions, as these are handled by the system.
Composite gestures: High-level spatial input
Both hand gestures and motion controllers can be tracked over time to detect a common set of high-level composite gestures. This enables your app to detect high-level tap, hold, manipulation and navigation gestures, whether users end up using hands or controllers.
Rendering the motion controller model
Windows makes available to apps a renderable model of each motion controller currently active in the system. By having your app dynamically load and articulate these system-provided controller models at runtime, you can ensure your app is forward-compatible to any future controller designs.
These renderable models should all be rendered at the grip pose of the controller, as the origin of the model is aligned with this point in the physical world. If you are rendering controller models, you may then wish to raycast into your scene from the pointer pose, which represents the ray along which users will naturally expect to point, given that controller's physical design.
Q: Controller is stuck in an infinite reboot (buzzing after LEDs cycle) OR the controller doesn't turn on at all
A: Verify you have fresh batteries in the device. If the issue persists, perform device recovery to reset the controller back to factory settings.
Q: I’m trying to pair my controllers, but they never show up in the “Add a new device menu” in Bluetooth Settings.
Q: Mixed Reality Portal is working, but motion controllers do not appear OR appear but track very badly (controllers keep flying away, shaking etc.)
A: Check the Bluetooth Best Practices section below. Both these symptoms are generally caused by Bluetooth ending up in a bad state.
Bluetooth best practices
Motion Controllers use the same Bluetooth technology found in many consumer devices. Motion Controllers are designed to work with Bluetooth capability included in any recent PC.
First verify that your PC has a Bluetooth radio. If the device passed the Mixed Reality Compatibility Checker then it should. Right click on the Windows Start Menu and select Device Manager. Expand the Bluetooth section and look for an Adapter.
Screenshot of an Example Device Manager. The Adapter is the Bluetooth radio.
If your computer has Bluetooth already, but you are still having problems with the Motion Controllers, consider replacing your Bluetooth radio with the Plugable external Bluetooth Adapter plugged into USB.
Note: you can only have one Bluetooth radio adapter active at a time. if you plug in an external radio in addition to an existing radio you need to disable your existing Bluetooth radio in Device Manager (right click on the adapter and click Disable Device) and un-pair / re-pair all of your previous Bluetooth devices.
Motion Controllers should work with other Bluetooth keyboards, mice and game controllers, but the experience will vary depending on the model of keyboard, mouse or game controller you use.
If you're using a USB Bluetooth Adapter:
Please ensure the USB Bluetooth Adapter is connected to a USB 2.0 port (black and doesn’t say SS) if available
Please plug the USB Bluetooth Adapter into a port that is physically separated from:
the HMD USB connector,
or wireless USB receivers like those for keyboards/mice.
Plugging the USB Bluetooth Adapter into the opposite side of the computer as far as possible from these other connectors is ideal.
Close the Bluetooth settings window if it's open. Leaving it open in the background means that a lot of extra calls are made to the Bluetooth protocol.
If you are using an internal Bluetooth card, please ensure you are using an external Bluetooth antenna. No external Bluetooth antenna in this case is known to cause tracking issues. If this doesn’t work, please use an external Bluetooth dongle (USB) after disabling the internal Bluetooth.
It is important that the device appears under Mouse, Keyboard & Pen category in the Bluetooth settings. If under Other devices then unpair/pair.
Please remove, un-pair and power off Bluetooth headphones and speakers. These are not supported with Windows Mixed Reality. You can use the headphone jack or built-in speakers on your Mixed Reality headset for the best audio experience.
If the controller isn’t working correctly and you’re unable to update the device, you can always restore the device to factory conditions. To recover your device:
Unplug and power off the controllers.
Open the battery cover.
Insert batteries (ensure you have good batteries when performing device recovery).
Press and hold pairing button (tab at the bottom under the batteries).
While holding pairing button, power on the controller by pressing and holding the Windows button for 5 seconds (keep both buttons held).
Release buttons and wait for controller to power on. This takes up to 15 seconds and there are no indicators when device recovery is happening. If device powers on immediately on button release, recovery button sequence did not get registered and you need to try again.
Remove old controller associations from Bluetooth settings (Settings->Bluetooth & other devices, select "Motion controller - Left" or "Motion controller - Right" and Remove device. After that pair controller with PC again.
After connecting with host and HMD, device will update firmware to latest available.
After firmware has been updated you need to unpair and pair controller again to get it working properly.
Lights and indicators
Motion controller uses LED constellation ring and haptics for indicating about its state.
Motion controller state
How you get into this state
Motion controller light and vibration behavior associated with state
Press and hold Windows button on controller for 2s to turn on controller.
LEDs turn on and controller vibrates once.
Press and hold Windows button on controller for 4s to turn off controller.
LEDs turn off and controller vibrates twice.
Controller enters sleeping state automatically when it’s motionless for 30s.
Controller automatically wakes when it detects motion (except when device is not paired with host PC, button press will be required to wake-up).
LEDs turn off, blink every 3 seconds while in sleeping state.
Press and hold pairing button inside battery case for 3s.
LEDs slowly pulse while in pairing mode.
LEDs go solid when exiting pairing mode. Controller vibrates once if pairing was successful or vibrates 3 times if pairing is unsuccessful and times out.
Controller connects to/disconnects from PC
Controller successfully connects to PC after you turned it on.
Controller disconnects from PC during use for some reason.
Controller vibrates once on PC connection or disconnection.
Low Battery Level
When battery level is low.
No LED or vibration indication when battery is low. If you look at the representation of the controller in headset, there is a battery indicator icon on the handle. When battery is low, the indicator icon will show 1/4 full.
Critical Battery Level
During power on when battery level is "Critical". “Critical” battery level means there is insufficient power for controller to stay on and the controller will turn off automatically.
Controller vibrates 3 times when you turn it on, then automatically turns off. As you approach this state, the battery indicator icon will display red.
This happens when you turn the controller on and a firmware update is required.
During controller firmware update, the controller will be unresponsive, though you will still be able to turn it off. If you turn it off during firmware update, the controller will simply try to update again the next time you turn it on and it connects to the PC.
When a controller firmware update starts, you’ll see a Windows notification telling you that the controller is being updated.
During firmware update, the LEDs will blink in an alternating pattern of half of the outer ring lights, then half of the inner ring lights