Cortana is a personal digital assistant working across all your devices to help you in your daily life. She learns about you; helps you get things done by completing tasks; interacts with you using natural language in a consistent, contextual way; and always looks out for you. Cortana has a consistent visual identity, personality, and voice.
IoT Core is an edition of Windows 10, and is optimized for small footprint and low cost IoT devices. Cortana is enabled on IoT Core in the Windows 10 Creators Update.
To use Cortana:
The device must have internet connection.
The user is required to sign in using their MSA on the device to utilize Cortana.
The device must have a display.
A microphone and speaker are required for speech interaction with Cortana.
An OEM must follow the guidance provided for the design and development of audio input devices outlined in the Microsoft Speech Platform Specification.
This document describes how to enable and utilize Cortana on IoT Core. Makers and OEMs can now leverage the capabilities of Cortana to build even smarter, connected IoT devices.
Windows 10 IoT Core can be run on a list of IoT devices.
Any microphone and speaker you select to use with the your IoT devices should work with Cortana. For better speech recognition quality, here is a recommended list of hardware that has been tested.
Blue Microphones Snowball iCE Condenser Microphone, Cardioid
Sound Tech CM-1000USB Table Top Conference Meeting Microphone with Omni-Directional Stereo
Microsoft LifeCam HD 3000
To get started with Cortana on Windows 10 IoT Core, a few additional set up steps are necessary.
Once the image boots up, please open Device Portal for your device and install updates. To do this, enter http://<device IP>:8080/#Windows%20Update in a browser and click on Check for Updates. Apply any updates if they are available. The update process will take approximately 30-40 minutes. Once the updates have been downloaded and installed, click on Restart Now.
Connect the microphone and speakers into the USB port on your device. If necessary, use a USB hub.
Once connected, adjust the microphone and speaker settings in Device Portal. To do this, enter http://<device IP>:8080/#Device%20Settings in a browser. Under Audio Control, check that the microphone and speakers displayed are the ones that are physically connected. In the image below, the speakers show Speakers (2- USB AUDIO) which is the Logitech USB Speakers and the microphone shows Desktop Microphone (Microsoft LifeCam HD-3000) which is the Microsoft LifeCam HD 3000.
Adjust the volume settings for both to be within the range of 40-70% (Double-check that the Microphone setting is not 0.0)
NOTE: This additional step is only for Dragonboard 410c.
To enable USB audio, you will need to disable the Qualcomm audio driver. To do this, simply run this command in a PowerShell window under the IoT Device administrator account:
devcon disable “AUDD\QCOM2468”
Now it’s time to launch Cortana!
During the first boot, right after Wi-Fi setting, Cortana consent will pop up to ask for permission. If you deny consent, Cortana will not work. To accept, click Sure, Cortana will be launched when you say “Hey Cortana”.
If you skip the acceptance, you need to go to Device Portal to enable Cortana later.
MSA sign in will pop up after consent. If you’d like to sign in, follow her instructions on the sign in page.
Cortana should work now if you accept the Consent.
If Cortana still doesn’t work, please do the following instructions to turn on Cortana manually.
Enter Device Portal again - to do this, enter http://<device IP>:8080/#Device%20Settings into a browser. Under Device Settings, scroll to the bottom and check “Start Cortana on Boot” if it is not checked. Restart the device (top right corner of the browser has a Power button with Restart option)
Go to Apps Manager under Device Portal - to do this, enter http://<device IP>:8080/#Apps%20manager into a browser. Under the list of App Names, you should see Search or Cortana. If it’s stopped, start the app (under the Actions drop down box, select Start). Cortana will launch on your IoT Device
If it’s your very first time launching Cortana, it will ask for consent. If you deny consent, Cortana will not work. To accept, click Sure:
If you deny consent at first and want to use Cortana later, you could go to Device Settings in the IoTCoreDefaultApp to turn on Cortana.
Signing in with your MSA will help Cortana provide more personalized context in her responses to you. If you deny log in, Cortana will still work but her responses will just be more generalized.
If you’d like to sign in, follow her instructions on the sign in page.
Sign in page will only be launched at the first time you ask Cortana to do any personal information related action. If you click ‘Maybe later’, it will be popped up next time when you ask personal information related question.
If you want to sign out your MSA, please go to Device Settings in the IoTCoreDefaultApp, click ‘About Me’, then the account icon at the bottom to sign out.
You can now try Cortana!
To start Cortana, say “Hey Cortana” to your microphone. Cortana on IoT Core will now launch and you can start to talk to her. Here are some sample queries:
Hey Cortana, What’s the weather today?
Hey Cortana, what is the traffic in Seattle?
Hey Cortana, what is the stock price for Microsoft?
If you stop talking to Cortana, she will be dismissed. To start her up again, say “Hey Cortana” followed by your query. To stop Cortana, say “Hey Cortana, stop” to your microphone.
Cortana can be your personal assistant - to do so, you just need to sign-in with your MSA. If you declined to sign in when first launching Cortana, you can sign in at any time by asking Cortana a personal question. She will ask you to sign-in with your MSA.
For example, “Hey Cortana, what is the traffic to my home?” Cortana will bring up the Sign In page. Follow her instructions for sign in and you will be all set up to go!
As a OEM, if you want to build Cortana enabled products, please follow the instructions below.
Use the IoTSettings tool to change region, user language or speech language. This is a command line tool that can be invoked from an application using the ProcessLauncher API. These commands must be run as default account, not administrator.
-list uilanguage => list all UI languages
-list speechlanguage => list all speech languages
-get uilanguage => display current UI language
-get speechlanguage => display current speech language
-get region => display current region
-set uilanguage language_tag - (e.g.: fr-CA => set default UI language French Canadian)
-set speechlanguage language_tag - (e.g.: fr-CA => set speech language French Canadian)
-set region region_code - (e.g.: CA => set default region to Canada)
e.g.: “IoTSettings -list uiLanguage” will give back the list of supported UI language (in the version of Windows IoT core image it has been executed against)
Note that Cortana will only work when region, UI language and speech language are coherent, e.g.: region = CA, UI language = en-CA and speech language = en-CA.
There is one feature ID for Cortana, <Feature>IOT_CORTANA</Feature>, OEM needs to add this feature ID in their OEMInput XML.
Add <Feature>IOT_CORTANA_OBSCURELAUNCH<Feature> in OEMInput XML.
OEM should add the following snippet into their own code to make sure that consent will be launched before the user uses Cortana.
Keyword spotting, it is software keyword spotter which detects when the user says “Hey, Cortana” .
OEM has the flexibility to decide when to enable KWS. For example, OEM wants to enable KWS only when proximity sensor detects someone is nearby.
OEM will be able to set whether Cortana can be activated by voice (listen to “Hey Cortana”). These API will only be available in Embedded Mode to UWP applications.
Embedded mode is a restricted device mode that enables a device to gain access to features and APIs that are otherwise restricted in UWP, including:
Background applications (aka CBT or headless apps)
Use of the lowLevelDevice capability APIs
Use of systemManagement capability APIs
The Windows.Services.Cortana.CortanaSettings will provide the following
An API to check if Cortana is available
An API to check if user has consent to voice activation for Cortana
an API to control Cortana voice activation (listening to “Hey Cortana”).
IoT OEM has a UWP app that enables voice activation (Cortana can listen to “Hey Cortana”) when user is close to a device. The sample below shows how to do that.
Cortana on IoT Core offers reactive experiences in the Windows 10 Creators Update. A reactive experience occurs when the user initiates commands to cut through multiple steps to complete a task. Reactive information is provided in response to a query.
Cortana is extensible. In addition to the native skills supported by the Cortana application, developers can create their own skills to allow Cortana to do more.
With the Windows 10 Creators Update, the following capabilities of Cortana will be enabled on IoT Core.
Catered to devices with small- or medium-sized screens (e.g. thermostat or refrigerator), provide a voice response with optimized visual content.
Leverages the audio pipeline provided in the Windows 10 operating system which supports linear microphone arrays. Audio input devices should conform to the guidance outlined in theMicrosoft Speech Platform.
To wake-up Cortana the user says “Hey, Cortana.” Keyword Spotting (KWS) runs locally to receive the voice input and complete the analysis. The audio is only sent to the cloud once the keyword is spotted. User consent is needed before enabling KWS. The KWS is optimized by the Windows Speech Platform and supports multiple languages and regions.
Will support en-US only.
Cortana custom skill provides the extensible capability for Cortana. The experts control the end-to-end experience, while Cortana brokers to relevant applications, websites, services and bots. Custom skills are created by developers, for example, OEM partners or ISVs.
OEMs can write a Voice Command Definition application that allows to add local commands to Cortana.