Mixed reality

Mixed reality is the result of blending the physical world with the digital world. Mixed reality is the next evolution in human, computer, and environment interaction and unlocks possibilities that before now were restricted to our imaginations. It is made possible by advancements in computer vision, graphical processing power, display technology, and input systems. The term mixed reality was originally introduced in a 1994 paper by Paul Milgram and Fumio Kishino, "A Taxonomy of Mixed Reality Visual Displays." Their paper introduced the concept of the virtuality continuum and focused on how the categorization of taxonomy applied to displays. Since then, the application of mixed reality goes beyond displays but also includes environmental input, spatial sound, and location.

Environmental input and perception

Advancements in sensors and processing are giving rise to a new area of computer input from environments. The interaction between computers and environments is effectively environmental understanding, or perception. Hence the API names in Windows that reveal environmental information are called the perception APIs. Environmental input captures things like a person's position in the world (e.g. head tracking), surfaces and boundaries (e.g. spatial mapping and spatial understanding), ambient lighting, environmental sound, object recognition, and location.

Now, the combination of all three – computer processing, human input, and environmental input – sets the opportunity to create true mixed reality experiences. Movement through the physical world can translate to movement in the digital world. Boundaries in the physical world can influence application experiences, such as game play, in the digital world. Without environmental input, experiences cannot blend between the physical and digital realities.

The mixed reality spectrum

Since mixed reality is the blending of the physical world and digital world, these two realities define the polar ends of a spectrum known as the virtuality continuum. For simplicity, we refer to this as the mixed reality spectrum. On the left-hand side we have physical reality in which we, humans, exist. Then on the right-hand side we have the corresponding digital reality.

Most mobile phones on the market today have little to no environmental understanding capabilities. Thus the experiences they offer cannot mix between physical and digital realities. The experiences that overlay graphics on video streams of the physical world are augmented reality, and the experiences that occlude your view to present a digital experience are virtual reality. As you can see, the experiences enabled between these two extremes is mixed reality:

• Starting with the physical world, placing a digital object, such as a hologram, as if it was really there.
• Starting with the physical world, a digital representation of another person – an avatar – shows the location where they were standing when leaving notes. In other words, experiences that represent asynchronous collaboration at different points in time.
• Starting with a digital world, physical boundaries from the physical world, such as walls and furniture, appear digitally within the experience to help users avoid physical objects.

Most augmented reality and virtual reality offerings available today represent a very small part of this spectrum. They are, however, subsets of the larger mixed reality spectrum. Windows 10 is built with the entire spectrum in mind, and allows blending digital representations of people, places and things with the real world.

There are two main types of devices that deliver Windows Mixed Reality experiences:

1. Holographic devices. These are characterized by the device's ability to place digital content in the real world as if it were really there.
2. Immersive devices. These are characterized by the device's ability to create a sense of "presence" – hiding the physical world and replacing it with a digital experience.

Characteristic	Holographic Devices	Immersive Devices
Example Device	Microsoft HoloLens	Acer Windows Mixed Reality Development Edition
Display	See-through display. Allows user to see physical environment while wearing the headset.	Opaque display. Blocks out the physical environment while wearing the headset.
Movement	Full six-degrees-of-freedom movement, both rotation and translation.	Full six-degrees-of-freedom movement, both rotation and translation.

Note, whether a device is connected to or tethered to a separate PC (via USB cable or Wi-Fi) or self-contained (untethered) does not reflect whether a device is holographic or immersive. Certainly, features that improve mobility lead to better experiences and both holographic and immersive devices could be tethered or untethered.

Devices and experiences

Technological advancement is what has enabled mixed reality experiences. There are no devices today that can run experiences across the entire spectrum; however, Windows 10 provides a common mixed reality platform for both device manufacturers and developers. Devices today can support a specific range within the mixed reality spectrum, and over time new devices should expand that range. In the future, holographic devices will become more immersive, and immersive devices will become more holographic.

Often, it is best to think what type of experience an app or game developer wants to create. The experiences will typically target a specific point or part on the spectrum. Then, developers should consider the capabilities of devices they want to target. For example, experiences that rely on the physical world will run best on HoloLens.

• Towards the left (near physical reality). Users remain present in their physical environment and are never made to believe they have left that environment.
• In the middle (fully mixed reality). These experiences perfectly blend the real world and the digital world. Viewers who have seen the movie Jumanji can reconcile how the physical structure of the house where the story took place was blended with a jungle environment.
• Towards the right (near digital reality). Users experience a completely digital environment and are oblivious to what occurs in the physical environment around them.

Here's how different experiences take advantage of their position on the mixed reality spectrum:

• Skype on Microsoft HoloLens. This experience allows collaboration through drawing in someone's physical environment. As an experience, it is currently further left on the spectrum because the physical environment remains the location of activity.
• Fragments and RoboRaid. Both of these take advantage of the layout of the user's physical environment, walls, floors, furniture to place digital content in the world. The experience moves further to the right on the spectrum, but the user always believes they are in their physical world.
• HoloTour on Microsoft HoloLens. HoloTour is designed with an immersive experience in mind. Users are meant to walk around tourist locations. On HoloLens, HoloTour pushes the limits of the device's immersive capabilities.
• HoloTour on immersive devices. Meanwhile when HoloTour runs on an immersive device, it showcases the environmental input by allowing users to walk around the tourist location. The boundaries that help users avoid walking into walls represent further capabilities that pull the experience towards the middle.
• 360° videos. Since environmental input like translational movement does not affect the video playback experience, these experiences fall to the far right towards digital reality, effectively fitting into the narrow part of the spectrum that is virtual reality.

Skype for HoloLens, Fragments and RoboRaid are best experienced with HoloLens. Likewise, 360° video is best experienced on immersive devices. HoloTour showcases the best of what both types of devices can do today when trying to push towards the center experience of mixed reality.

See also

• API Reference: Windows.Perception
• API Reference: Windows.Perception.Spatial
• API Reference: Windows.Perception.Spatial.Surfaces