Holograms 210

Gaze is the first form of input and reveals the user's intent and awareness. Holograms 210 (aka Project Explorer) is a deep dive into gaze-related concepts for Windows Holographic. We will be adding contextual awareness to our cursor and holograms, taking full advantage of what your app knows about the user's gaze.

We have a friendly astronaut here to help you learn gaze concepts. In Holograms 101, we had a simple cursor that just followed your gaze. Today we're moving a step beyond the simple cursor:

  • We're making the cursor and our holograms gaze-aware: both will change based on where the user is looking - or where the user is not looking. This makes them context-aware.
  • We will add feedback to our cursor and holograms to give the user more context on what is being targeted. This feedback can be audio and visual.
  • We'll show you targeting techniques to help users hit smaller targets.
  • We'll show you how to draw the user's attention to your holograms with a directional indicator.
  • We'll teach you techniques to take your holograms with the user as she moves around in your world.

Prerequisites

Project files

  • Download the files required by the project.
  • Unarchive the files to your desktop or other easy to reach location.

Errata and Notes

  • "Just My Code" needs to be disabled (unchecked) under Tools->Options->Debugging in order to hit breakpoints in your code.

Chapter 1 - Unity Setup

Objectives

  • Optimize Unity for HoloLens development.
  • Import assets and setup the scene.
  • View the astronaut in the HoloLens.

Instructions

  • Start Unity.
  • Select New Project.
  • Name the project ModelExplorer.
  • Enter location as the Gaze folder you previously unarchived.
  • Make sure the project is set to 3D.
  • Click Create Project.

Unity settings for HoloLens

We need to let Unity know that the app we are trying to export should create a holographic view instead of a 2D view. We do that by adding HoloLens as a virtual reality device.

  • Go to Edit > Project Settings > Player.
  • In the Inspector Panel for Player Settings, select the Windows Store icon.
  • Expand the Other Settings group.
  • In the Rendering section, check the Virtual Reality Supported checkbox to add a new Virtual Reality SDKs list.
  • Verify that Windows Holographic appears in the list. If not, select the + button at the bottom of the list and choose Windows Holographic.
  • Go to Edit > Project Settings > Quality.
  • Click on downward pointing arrow in the Default row under the Windows Store icon.
  • Select Fastest for Windows Store Apps.

Import art assets

  • Right click the Assets folder in the Project panel.
  • Click on Import Package > Custom Package.
  • Navigate to the project files you downloaded and click on ModelExplorer.unitypackage.
  • Click Open.
  • After the package loads, click on the Import button.

Setup the scene

  • In the Hierarchy, delete the Main Camera.
  • In the HoloToolkit folder, open the Utilities folder, then open the Prefabs folder.
  • Drag and drop the Main Camera from the Prefabs folder into the Hierarchy.
  • Right-click the Directional Light in the Hierarchy and select Delete.
  • In the Holograms folder, drag and drop the following assets into the root of the Hierarchy:
    • AstroMan
    • Lights
    • SpaceAudioSource
    • SpaceBackground
  • Start Play Mode to view the astronaut.
  • Click Play Mode again to Stop.
  • In the Holograms folder, find the Fitbox asset and drag it to the root of the Hierarchy.
  • Select the Fitbox in the Hierarchy panel.
  • Drag the AstroMan collection to the Hologram Collection property of the Fitbox in the Inspector panel.

Build the project

  • Save the new scene: File > Save Scene As.
  • Click New Folder and name the folder Scenes.
  • Name the file “ModelExplorer” and save it in the Scenes folder.
  • In Unity select File > Build Settings.
  • Click Add Open Scenes to add the scene.
  • Select Windows Store in the Platform list and click Switch Platform.
  • Set SDK to Universal 10 and Build Type to D3D.
  • Check Unity C# Projects.
  • Click Build.
  • Create a New Folder named "App".
  • Single click the App Folder.
  • Press Select Folder.
  • When Unity is done, a File Explorer window will appear.
  • Open the App Folder.
  • Open the ModelExplorer Visual Studio Solution.
  • In Visual Studio, right click on Package.appxmanifest in the Solution Explorer and select View Code
  • Find the line specifying TargetDeviceFamily and change Name="Windows.Universal" to Name="Windows.Holographic".
  • Save the Package.appxmanifest.
  • Using the top toolbar in Visual Studio, change the target from Debug to Release and from ARM to x86.
  • Click on the drop down arrow next to the Device button, and select Remote Machine.
  • Enter your device IP address and set Authentication Mode to Universal (Unencrypted Protocol). Click Select. If you do not know your device IP address, look in Settings > Network & Internet > Advanced Options.
  • In the top menu bar, click Debug -> Start Without debugging or press Ctrl + F5. If this is the first time deploying to your device, you will need to pair it with Visual Studio.
  • When the app has deployed, dismiss the Fitbox with a select gesture.

Chapter 2 - Cursor and target feedback

Objectives

  • Cursor visual design and behavior.
  • Gaze-based cursor feedback.
  • Gaze-based hologram feedback.

We're going to base our work on some cursor design principles, namely:

  1. The cursor is always present.
  2. Don't let the cursor get too small or big.
  3. Avoid obstructing content.

Instructions

  • Click the Create menu on top of the Hierarchy panel.
  • Select Create Empty.
  • Right-click the new GameObject and rename it to "Managers".
  • In the Hierarchy panel, select the Managers object.
  • In the Inspector panel, click the Add Component button.
  • In the menu, type in the search box Gaze Manager. Select the search result.
  • In the Inspector panel, select the RaycastLayerMask dropdown and unselect TransparentFX.
  • In the HoloToolkit\Input\Prefabs folder, find the Cursor asset.
  • Drag and drop the Cursor onto the Hierarchy.
  • In the Hierarchy panel, select the Cursor object.
  • In the Inspector panel, click the Add Component button.
  • In the menu, type in the search box Cursor Manager. Select the search result.
  • Expand the Cursor object in the Hierarchy.
  • Drag and drop the CursorOnHolograms object onto the Cursor On Holograms property in the Inspector.
  • Drag and drop the CursorOffHolograms object onto the Cursor Off Holograms property in the Inspector.

At this point, you need to edit the GazeManager.cs file, and edit it to perform the following tasks:

  1. Perform a physics raycast,
  2. Store the position and normal of the raycast intersection,
  3. If the raycast did not hit anything, set the position and normal to defaults.

You can write the code yourself by locating comments in GazeManager.cs labeled "Coding Exercise" - each comment corresponds to one line of code. You can also use this completed version:

GazeManager.cs[show]

Next you will need to edit CusorManager.cs in order to achieve the following:

  1. Decide which cursor should be active,
  2. Update the cursor based on whether it's off or on a hologram,
  3. Position the cursor where the user is gazing.

Again, you are welcome to write your own code by locating the comments labeled "Coding Exercise" in CursorManager.cs, or use the following:

CursorManager.cs[show]

  • Rebuild the app from File > Build Settings.
  • Open the App Folder.
  • Open the ModelExplorer Visual Studio Solution.
  • Click Debug -> Start Without debugging or press Ctrl + F5.
  • Observe how the cursor is drawn, and how it changes appearance if it is touching a hologram.
  • In the Hierarchy panel, select the Managers object.
  • In the Inspector panel, click the Add Component button.
  • In the menu, type in the search box Interactible Manager. Select the search result.
  • In the Hierarchy panel, select the AstroMan object.
  • In the Inspector panel, click the Add Component button.
  • In the menu, type in the search box Interactible. Select the search result.

You'll need to edit both InteractibleManager.cs and Interactible.cs in order to achieve the following:

  1. In InteractibleManager.cs, fetch the point the gaze raycast hit and save the collided GameObject.
  2. Send the GazeEntered message if gazing at an object with which you can interact.
  3. Send the GazeExited message if gazing away from the object with which you can interact.
  4. Handle the GazeEntered and GazeExited callbacks in Interactible.cs.

Give it a try yourself with the coding exercises in InteractibleManager.cs and Interactible.cs, or use the following solutions:

InteractibleManager.cs[show]

Interactible.cs[show]

  • As before, build the project and deploy to the HoloLens.
  • Observe what happens when the gaze is aimed at an object, and when it's not.

Chapter 3 - Targeting Techniques

Objectives

  • Make it easier to target holograms.
  • Stabilize natural head movements.

Instructions

  1. In the Hierarchy panel, select the Managers object.
  2. In the Inspector panel, click the Add Component button.
  3. In the menu, type in the search box Gaze Stabilizer. Select the search result.

Now we'll modify our GazeManager to use the GazeStabilizer.

  1. Open the GazeManager script in Visual Studio.
  2. Copy the code below into GazeManager.cs, or complete coding exercise 3.a yourself.

GazeManager.cs[show]

Chapter 4 - Directional indicator

Objectives

  • Add a directional indicator on the cursor to help find holograms.

Instructions

We're going to use the DirectionalIndicator.cs file which will:

  1. Show the directional indicator if the user is not gazing at the holograms.
  2. Hide the directional indicator if the user is gazing at the holograms.
  3. Update the directional indicator to point to the holograms.

Let's get started.

  • Click on the AstroMan object in the Hierarchy panel and click the arrow to expand it.
  • In the Hierarchy panel, select the DirectionalIndicator object under AstroMan.
  • In the Inspector panel, click the Add Component button.
  • In the menu, type in the search box Direction Indicator. Select the search result.
  • In the Hierarchy panel, drag and drop the Cursor object onto the Cursor property in the Inspector.
  • In the Project panel, in the Holograms folder, drag and drop the DirectionalIndicator asset onto the Directional Indicator property in the Inspector.
  • Build and deploy the app to the HoloLens.
  • Watch how the direction indicator object helps you find the astronaut

Chapter 5 - Billboarding

Objectives

  • Use billboarding to have holograms always face towards you.

We will be using the Billboard.cs file to keep a GameObject oriented such that it is facing the user at all times.

  • In the Hierarchy panel, select the AstroMan object.
  • In the Inspector panel, click the Add Component button.
  • In the menu, type in the search box Billboard. Select the search result.
  • In the Inspector set the Pivot Axis to Y.
  • Try it! Build and deploy the app to the HoloLens as before.
  • See how the Billboard object faces you no matter how you change the viewpoint.
  • Delete the script from the AstroMan for now.

Chapter 6 - Tag-Along

Objectives

  • Use Tag-Along to have our holograms follow us around the room.

As we work on this issue, we'll be guided by the following design constraints:

  • Content should always be a glance away.
  • Content should not be in the way.
  • Head-locking content is uncomfortable.

The solution used here is to use a "tag-along" approach.

A tag-along object never fully leaves the user's view. You can think of the a tag-along as being an object attached to the user's head by rubber bands. As the user moves, the content will stay within an easy glance by sliding towards the edge of the view without completely leaving. When the user gazes towards the tag-along object, it comes more fully into view.

  • In the Hierarchy panel, select the Managers object.
  • In the Inspector panel, click the Add Component button.
  • In the menu, type in the search box Gesture Manager. Select the search result.

We're going to use the SimpleTagalong.cs file which will:

  1. Determine if the Tag-Along object is within the camera bounds.
  2. If not within the view frustum, position the Tag-Along to partially within the view frustum.
  3. Otherwise, position the Tag-Along to a default distance from the user.

To do this:

  • In the Holograms folder click on Tagalong asset.
  • In the Inspector, click on Tag dropdown and click Add Tag ….
  • Click on the + sign and name Tag 0 as TagAlong.
  • In the Holograms folder click on Tagalong asset and click on Tag dropdown.
  • Select TagAlong tag.
  • We first must change the Interactible.cs script to send a message to InteractibleAction.
  • Edit Interactible.cs by completing the coding exercise or use the following:

Interactible.cs[show]

The InteractibleAction.cs script performs custom actions when you gaze at the holograms. Let's update it for use with the tag-along.

  • In the Scripts folder click on InteractibleAction.cs asset to open in Visual Studio.
  • Complete the coding exercise or change it to this:
    • At the top of the Hierarchy, in the search bar type ChestButton_Center and select the result.
    • In the Inspector panel, click the Add Component button.
    • In the menu, type in the search box Interactible Action. Select the search result.
    • In Holograms folder find the Tagalong asset.
    • Select the ChestButton_Center object in the Hierarchy. Drag and drop the TagAlong object from the Project panel onto the Inspector into the Object to TagAlong property.
    • Double click the InteractibleAction script to open it in Visual Studio.

We need to add the following:

  • Add functionality to the PerformTagAlong function in the InteractibleAction script.
  • Add billboarding to the gazed-upon object, and set the pivot axis to free.
  • Then add simple Tag-Along to the object.

Here's our solution:

InteractibleAction.cs[show]

  • Try it! Build and deploy the app to the HoloLens.
  • Watch how the content follows the center of the gaze point, but not continuously and without blocking it.