Holograms 220

Spatial sound breathes life into holograms and gives them presence in our world. Holograms are composed of both light and sound, and if you happen to lose sight of your holograms, spatial sound can help you find them. Spatial sound is not like the typical sound that you would hear on the radio, it is sound that is positioned in 3D space. With spatial sound, you can make holograms sound like they're behind you, next to you, or even on your head! In this course, you will:

  • Configure your development environment to use Microsoft Spatial Sound.
  • Learn how to use spatial sound to ground holograms in the real world.
  • Use audio to find your holograms.
  • Use sound to provide haptic feedback when interacting with holograms.
  • Use sound to ground your hand in the holographic world.
  • Immerse yourself in music.

Prerequisites

Project files

  • Download the files required by the project.
  • Unarchive the files to your desktop or other easy to reach location.

Errata and Notes

  • "Enable Just My Code" needs to be disabled (unchecked) in Visual Studio under Tools->Options->Debugging in order to hit breakpoints in your code.

Chapter 1 - Unity Setup

Objectives

  • Change Unity's sound configuration to use Microsoft Spatial Sound.
  • Add 3D sound to an object in Unity.

Instructions

  • Download and unarchive the project files to your Desktop. Keep the folder name as Spatial Sound.
  • Start Unity.
  • Select Open.
  • Navigate to your Desktop and find the Spatial Sound folder you previously unarchived.
  • Click on the Starting folder and then press the Select Folder button.
  • Wait for the project to load in Unity.
  • In the Hierarchy panel, select the Main Camera.
  • In the Inspector panel, set the camera's position to 0, 0, 0.
  • In Unity's top menu, go to Edit > Project Settings > Audio.
  • Find the Spatializer Plugin dropdown, and select MS HRTF Spatializer.
  • In the Hierarchy panel, click Create > Create Empty.
  • Right-click on the new GameObject and rename it to “SpatialSound”.
  • With the SpatialSound object selected, in the Inspector panel, click Add Component, and select Audio > Audio Source
  • In the Inspector panel, find the Audio Source component.
  • Click the circle to the right of the AudioClip property.
  • In the dialogue window, find and select the Riff audio clip and then close the dialogue window.
  • In the Inspector panel, check the Spatialize checkbox.
  • Check the Loop checkbox.
  • Drag the Spatial Blend slider all the way to 3D, or enter 1 in the edit box.
  • Start Unity's play mode by pressing the Play button at the top, center of the editor.
  • In the Hierarchy panel, find and select the Main Camera object.
  • In the Inspector panel, hover over one of the Position variable names (X, Y, Z).
  • When you see two small arrows appear on either side of the cursor, click and drag your mouse to change the camera's positon.
  • As the camera changes position, you should notice that the audio changes too (if you move the camera left, the sound will be stronger on the right).
  • Click the Play button again to exit play mode.

Chapter 2 - Grounding Holograms

Objectives

  • Use the UAudioManager to play interactive sounds.
  • Experience the difference between stereo sound and Spatial Sound.

Instructions

  • In the Project panel, select the Scenes folder.
  • Double-click Grounding to open the scene in Unity.
  • In the Hierarchy panel, select the AudioEmitter object.
  • In the Inspector panel, find the Toggle Emitter component.
  • Double-click the Toggle Emitter script to open it in Visual Studio.

You can write the code yourself by locating comments in ToggleEmitter.cs labelled "Coding Exercise" - each comment corresponds to one line of code. You can also refer to this completed version:

ToggleEmitter.cs[show]

Build and Deploy

  • In Unity select File > Build Settings.
  • Click Add Open Scenes to add the scene.
  • Select Windows Store in the Platform list and click Switch Platform.
  • Set SDK to Universal 10 and Build Type to D3D.
  • Check Unity C# Projects.
  • Click Build.
  • Create a New Folder named "App".
  • Single click the App Folder.
  • Press Select Folder.
  • When Unity is done, a File Explorer window will appear.
  • Open the App Folder.
  • Double-click on UAudioTools.sln to load it in Visual Studio.
  • Using the top toolbar, change the target to Release and X86.
  • Click on the arrow to the right of 'Local Host', and select Remote Machine.
  • Enter your device's IP address and change Authentication Mode to Universal (Unencrypted Protocol).
  • Click Debug -> Start Without debugging or press Ctrl + F5.
  • Note, you might notice some red errors in the Visual Studio Errors panel. It is safe to ignore them. Errors in the Output panel will require you to make a fix (most often they are caused by a mistake in a script).
  • Once the app is deployed, observe the different types of sound emitted from the box.
    • The first select gesture performed on the box turns on stereo sound.
    • The second select gesture performed on the box turns on spatial sound.
    • The third select gesture performed on the box turns off sound.

Chapter 3 - Object Discovery

Objectives

  • Use Spatial Sound to find holograms in the world.

Instructions

  • In Unity's Project panel, select the Scenes folder.
  • Double-click ObjectDiscovery to open the scene in Unity.

Build and Deploy

  • As before, build the project in Unity and deploy in Visual Studio.
    • In the Build Settings window, uncheck Scenes/Grounding.
    • Click Add Open Scene to add the current scene
  • Say "start" to have the astronaut start talking.
  • Perform a select gesture on the astronaut. The astronaut will disappear and then move to a new location.
  • Try to find the astronaut based on the sounds coming from his location.

Chapter 4 - Audio Haptics

Objectives

  • Use sounds to replace tactile feedback of physical objects.
  • Use sound to ground your hand in the holographic world.

Instructions

  • In the Project panel, select the Scenes folder.
  • Double-click AudioHaptics to open the scene in Unity.

Build and Deploy

  • As before, build the project in Unity and deploy in Visual Studio.
    • In the Build Settings window, uncheck Scenes/ObjectDiscovery.
    • Click Add Open Scene to add the current scene
  • Move your hand into view and hold your index finger up. Listen to the sound that plays whenever your hand is being tracked. This is one example of how audio can be used as a haptic feedback mechanism.
  • Move your hand out of view and listen for the tracking lost sound.
  • Gaze at the cube closest to you and select it. Listen for the confirmation sound.
  • Gaze at the cube furthest from you and select it. The confirmation sound should be softer.
  • Perform a select gesture when nothing has focus. A different sound plays to let the user know that nothing was selected. This is another great example of audio haptic feedback at work!

Chapter 5 - Immersion

Objectives

  • Compare a musical experience in stereo and spatial sound.
  • Toggle instruments and move around to change the mix.

Instructions

  • In the Project panel select the Scenes folder.
  • Double-click Immersion to open the scene in Unity.

Build and Deploy

  • As before, build the project in Unity and deploy in Visual Studio. Be sure to remove the previous scene and add the current scene.
  • In the Build Settings window, uncheck Scenes/AudioHaptics.
  • Click Add Open Scene to add the current scene
  • Say "start" to start playing music with stereo sound. The spheres should be blue.
  • Say "spatial" to change the music to use spatial sound. The spheres should be red.
    • In spatial mode, you can toggle which instruments play by selecting the spheres.
    • Disabled instruments will be grey.
  • Say "standard" to have music play in stereo sound.
  • Walk around the room and switch between standard and spatial sound to compare them. Stereo sound always sounds the same no matter where you are, but with spatial sound, you know where each instrument is and can really feel like a part of the band.

The End

Congratulations! You have now completed Holograms 220 - Spatial Sound.

  • You know how to spatialize sound in Unity.
  • You were able to locate a hologram by following the spatial sound clues.
  • You heard several examples of audio being used as a haptic feedback mechanism on HoloLens.
  • With spatial sound, you changed an ordinary musical experience into extraordinary!