Designing for mixed reality

Designing holograms has been a fun and unique challenge for our team. Because we are learning about how to make great apps right alongside you, we don't have any hard and fast "rules". We've started writing down some things we found useful to share with each other and now you. As you discover design considerations you like while building apps, please jump into the forums, share with us and we'll update the docs here with your learning.

Basics of Hologram Rendering

We have made it a habit to render holograms for both eyes. Rendering a heads-up display in one eye only can make an interface hard to understand, as well as causing uneasiness to the user's eye and brain.

The User is the Camera

Always think about design for your user's unique point of view as they move about the world.

  • Is the user sitting, reclining, standing, or walking while using your experience?
  • How does your content adjust to different positions?
  • Can the user adjust it?
  • Will the user be comfortable using your app?

The user is the camera and they control the movement. Let them drive. Don't shake the camera, it can make them feel uncomfortable. Avoid abrupt movement. If you need to bring content to or from the user, move it slowly and smoothly toward them for maximum comfort. Users are sensitive to acceleration (both angular and translational). Do not accelerate or turn the user's camera.

Users will react to large menus coming at them.

  • Use shorter animations
  • Animate from down/left/right or fade in instead of Z
  • Slow down timing
  • Allow user to see the world in the background

Leverage the User's Holographic Frame

In 2D development, frequently accessed content and settings may be placed in the corners to make them easily accessible. However, in holographic apps, the corners of the holographic frame may be uncomfortable for the user to access. In this case, the center of the holographic frame is the prime location for content.

The user may need to be guided to help locate important events or objects outside the frame. You can use arrows, light trails, character head movement, thought bubbles, pointers, spatial sound, and voice prompts to help guide the user to important content in your app.

It is recommended to not lock content to the screen for the user's comfort. If you need to keep content in view, place it in the world and make the content "tag-along" like the Start Menu in the HoloLens Shell. Content that gets pulled along with the user's holographic frame will feel more natural in the environment.

Holograms feel real when they fit within the frame since they don't get cut off. Users will move in order to see the bounds of a hologram within the frame.

Simplify your UI to fit within the frame and keep your focus on the main action.

Share Your Experience

Using mixed reality capture, users can capture a photo or video of their experience at any time. Consider experiences in your app where you may want to encourage snapshots or videos.

Basic Hologram Positioning

  • The magic of HoloLens is in its ability to place content within the real world. This kind of "world-locked" content is by far one of the biggest and most exciting areas to explore while developing your app.
  • Some scenarios may demand that holograms remain easily discoverable or visible throughout the experience. There are two high-level approaches to this kind of positioning. Let's call them "display-locked" and "body-locked".
    • Display-locked content is positionally "locked" to the device display. This is tricky for a number of reasons, including an unnatural feeling of "clingyness" that makes many users frustrated and wanting to "shake it off." In general, many designers have found it better to avoid display-locking content.
    • The body-locked approach is far more forgivable. Body-locking is when a hologram is tethered to the user's body or gaze vector, but is positioned in 3d space around the user. Many experiences have adopted a body-locking behavior where the hologram "follows" the users gaze, which allows the user to rotate their body and move through space without losing the hologram. Incorporating a delay helps the hologram movement feel more natural. For example, some core UI of the Windows Holographic OS uses a variation on body-locking that follows the user's gaze with a gentle, elastic-like delay while the user turns their head.
  • Place the hologram at a comfortable viewing distance typically about 1-2 meters away from the head.
  • Provide an amount of drift for elements that must be continually in the holographic frame, or consider animating your content to one side of the display when the user changes their point of view.

Embed Your Holograms in the World

Use spatial mapping to place objects on surfaces. This helps anchor objects in the user's world and takes advantage of real world depth cues. Holograms floating in space or moving with the user will not feel as real. When possible, place items for comfort.

Respect real world depth ordering. Occluding your holograms based on other holograms and real world objects helps convince the user that these holograms are actually in their space.

Visualize surfaces when placing or moving holograms (use a simple projected grid). This will help the user know where they can best place their holograms, and shows the user if the spot they are trying to place the hologram hasn't been mapped yet. You can "billboard items" toward the user if they end up at too much of an angle.

Optimal distance for placing holograms from the user.
Optimal distance for placing holograms from the user.

Place holograms in the optimal zone - between 1.25m and 5m. Two meters is the most optimal, and the experience will degrade the closer you get from one meter. At distances nearer than one meter, holograms that regularly move in depth are more likely to be problematic than stationary holograms. Consider ways of gracefully clipping or fading out your content when it gets too close so as not to jar the user into an unexpected experience.

Integrating Holograms and the Real World

  • Aligning to gravitational rules makes holograms easier to relate to and more believable. eg: Place a holographic dog on the ground & a vase on the table rather than have them floating in space.
  • Many designers have found that they can even more believably integrate holograms by creating a "negative shadow" on the surface that the hologram is sitting on. They do this by creating a soft glow on the ground around the hologram and then subtracting the "shadow" from the glow. The soft glow integrates with the light from the real world and the shadow grounds the hologram in the environment.