Event - Computer Vision on the HoloLens at ICCV 2017

Tutorial: Computer Vision on Microsoft HoloLens
October 22nd, 2017
ICCV 2017 Conference


  • Marc Pollefeys
  • Georg Klein
  • Pawel Olszta
  • Sudipta Sinha


Microsoft released the HoloLens Developer Edition in March 2016. It is the world’s first untethered augmented-reality device. Developers can create “holographic” applications and share them with the user community through Microsoft’s App Store.

Internally, the HoloLens uses state-of-the-art computer vision algorithms for robust head-tracking, depth calculation, hand tracking and surface reconstruction.

The HoloLens is also a potent research device. Application code can access audio and video streams and surface meshes, all in a world coordinate space maintained by HoloLens’ highly accurate head-tracking.

This half day tutorial at ICCV 2017 we will dive into the vision processing on the HoloLens and show how to build applications that access the sensor streams. We will also be demonstrating the latest updates to the HoloLens software which offer increasing access to the full power of the device.


The tutorial will cover:

  • The architecture of the HoloLens.
  • How the heavy lifting of head-tracking, hand-tracking, and surface reconstruction are accomplished while leaving lots of CPU/GPU headroom for applications.
  • What sensors are contained in the HoloLens, and how they are used by standard applications.
  • How simple apps are built and deployed to the device.
  • How to access streams from the video camera, the microphone, and the surface mesh pipeline.
  • How to use HoloLens head-tracking to establish a 6DOF position for every sample.
  • Sample code for moving data streams off-device for processing on a desktop or in the cloud.
  • Latest news on surfacing other streams from the device for research purposes.

Attendees to this tutorial will leave with a good sense of how the HoloLens can be used for a range of Computer Vision research tasks, and materials to quickly get them started using the device.