Since a user could take a mixed reality capture (MRC) at any time, there are a few things that you should keep in mind when developing your application. This includes best practices for MRC visual quality and being responsive to system changes while MRCs are being captured.
Mixed reality captured photos and videos are likely the first exposure a user will have of your app. Whether as holographic screenshots on your Windows Store page or from other users sharing MRCs on social networks. You can use MRC to demo your app, educate users, encourage users to share their mixed world interactions, and for user research and problem solving.
By default, an app does not have to do anything to enable users to take mixed reality captures.
When a 2D app uses DXGI_PRESENT_RESTRICT_TO_OUTPUT or DXGI_SWAP_CHAIN_FLAG_HW_PROTECTED to show protected content with a properly-configured swap chain, the app's visual content will be automatically obscured while mixed reality capture is running.
The AppCapture class can be used by an app to know when system mixed reality capture is running (for either audio or video).
Note: AppCapture's GetForCurrentView API can return null if mixed reality capture isn't available on the device. It's also important to de-register the CapturingChanged event when your app is suspended, otherwise MRC can get into a blocked state.
MRC is expected to work without additional work from developers, but there are a few things to be aware of to provide the best mixed reality capture experience of your app.
MRC uses the hologram’s alpha channel to blend with the camera imagery
The most important step is to make sure your app is clearing to transparent black instead of clearing to opaque black. In Unity, this is done by default with the HoloToolkit but if you are developing in non-Unity, you may need to make a one line change.
Here are some of the artifacts you might see in MRC if your app is not clearing to transparent black:
Example Failures: Black edges around the content
Example Failures: The entire background scene of the hologram appears black
Expected Result: Holograms appear properly blended with the real-world
You can now tune the alpha values of your assets if you’d like, but typically don’t need to. Most of the time, MRCs will look good out of the box. MRC assumes pre-multiplied alpha. The alpha values will only affect the MRC capture.
The HoloLens Emulator does not have a photo video camera, so the real world is replaced by black. As a result, an MRC taken in the emulator won't expose the rendering issues mentioned above.
The mixed reality capture page of the Windows Device Portal has support for:
While the user can always trigger a photo or video using the system MRC capture service, an application may want to build a custom camera app but include holograms in the camera stream just like MRC. This allows the application to kick off captures on behalf of the user, build custom recording UI, or customize MRC settings to name a few examples.
Unity Applications should see Locatable_camera_in_Unity for the property to enable holograms.
Other applications can do this by using the Windows Media Capture APIs to control the Camera and add an MRC Video and Audio effect to include virtual holograms and application audio in stills and videos.
Applications have two options to add the effect:
(Note that these namespaces will not be recognized by Visual Studio, but the strings are still valid)
MRC Video Effect (Windows.Media.MixedRealityCapture.MixedRealityCaptureVideoEffect)
|Property Name||Type||Default Value||Description|
|StreamType||UINT32 (MediaStreamType)||1 (VideoRecord)|| Describe which capture stream this effect is used for.
Audio is not available.
|HologramCompositionEnabled||boolean||TRUE||Flag to enable or disable holograms in video capture.|
|RecordingIndicatorEnabled||boolean||TRUE||Flag to enable or disable recording indicator on screen during hologram capturing.|
|VideoStabilizationEnabled||boolean||FALSE||Flag to enable or disable video stabilization powered by the HoloLens tracker.|
|VideoStabilizationBufferLength||UINT32||0|| Set how many historical frames are used for video stabilization.
0 is 0-latency and nearly "free" from a power and performance perspective. 15 is recommended for highest quality (at the cost of 15 frames of latency and memory).
|GlobalOpacityCoefficient||float||0.9||Set global opacity coefficient of hologram in range from 0.0 (fully transparent) to 1.0 (fully opaque).|
MRC Audio Effect (Windows.Media.MixedRealityCapture.MixedRealityCaptureAudioEffect)
|Property Name||Type||Default Value||Description|
|MixerMode||UINT32||2|| 0 : Mic audio only
1 : System audio only
2 : Mic and System audio
There are certain limitations around multiple apps accessing MRC at the same time.
The photo/video camera can only be accessed by a single process at a time. While a process is recording video or taking a photo any other process will fail to acquire the photo/video camera. (this applies to both Mixed Reality Capture and standard photo/video capture)
An app's custom MRC recorder is mutually exclusive with system MRC (capturing photos, capturing videos, or streaming from the Windows Device Portal).
When MRC is enabled: